[S] Multiple "comparisons"

Tue, 24 Feb 1998 01:33:37 -0500 (EST)

Can someone tell me if the problem of "multiple comparison" would apply to testing of the same hypothesis repeatedly in correlation anlaysis? By multiple comparison I mean the problem that alpha levels are inflated if you conduct several tests of one hypothesis independently

My research hypothesis is that two variables are significantly correlated.. One variable is always the same variable. The problem is that I have multiple measures of the second variable, but that testing of each of these is a test of the research hypothesis.

1. The research hypothesis is that there will be significant correlation between a variable and another.

2. Let's say for example that the first variable is temperature and I want to know whether the frequency with which items are bought together in pairs varies with temperature. The second variable is a coefficient of association (phi, a measure of dichotomous association) between two items. I have numerous pairwise combinations of variables for which I have these association values. For example, say I have sunscreen and bandaids, sunscreen and snowshovels, Pepsi and asparagus, etc. Therefore I am testing for correlation between these pairwise association values for the items with the first variable, temperature.

3. I test whether the association value for Pepsi and asparagus is significantly associated with temperature. Then I test similarly for snowshovels and sunscreen with temperature, etc.

4. When I test the null, it is tested once for each pairwise combination. Are these separate tests or multiple "comparisons" (tests) of the same hypothesis? If I am viewing this correctly, I think I am testing the null of the research hypothesis (that there is no correlation between the first variable and the association values for the pairs) numerous times, although it is tested for correlation of temperature with a different pair of items each time (??). Is this correct?

5. Because of this problem of multiple "comparisons", my alpha is affected. If I assign an alpha of .05 to testing the null of the research hypothesis, 5% of my tests should falsely indicate that there is a significant correlation. Therefore if I test the hypothesis 100 times, I can expect that 5 of my tests will be falsely significant.

a) Is my reasoning correct? Is this multiple testing actually a problem? Or can I just take each test at its own significance level? Does the fact that the pairs are different make a difference?

b) How do I deal with this problem if it is in fact a problem ? I understand that the Bonferroni test can be applied by just multiplying the p for each hypothesis test by the number of tests to get the "real" significance level for each test, but that it is extremely conservative. To me this would mean that, to assign an alpha of .05 to the research hypothesis,, I need to conduct each of the 100 tests at the level of (0.05/100) = 0.0005 ! Is this correct?

c) How can I apply the Scheffe test to tests for significant correlations like this?

Frank O'Hare
Get free personalized email at http://www.lycosemail.com
This message was distributed by s-news@wubios.wustl.edu. To unsubscribe
send e-mail to s-news-request@wubios.wustl.edu with the BODY of the
message: unsubscribe s-news