I have already asked a similar question but did not explain my problem in entirety. Any help would be greatly appreciated!

I am analyzing some data to which I think I need to make some kind of multiple comparisons correction. The data comes from an economic game (Dictator game) and is the form of donations made by subjects ranging from $ 0.00 to $ 0.50 (can be any $ 0.01 denomination). Subjects were primed with an experimental image or an control image I am primarily interested in the effect of which image they were shown on their donation.

I did the following tests all on the same donation data; 1) I used a Mann-Whitney U test to test if there was a difference in the amount donated for each image shown, control or experimental prime (Main hypotheseis). 2)I tested If there was a difference in the amount donated by each gender (Mann-Whitney U test) 3)I tested the relationship between age of dictator and donation using Spearmans Rank 4) Income and Education were categorical, eg. one level of Education would be "Some High School" and one level of Income would be "$0-$15 000" for each there was 6 levels. I tested donation against the different levels of Income (Kruscall-Wallis) and did the same for Education (Kruscall-Wallis). 5)Although dictators could choose any value to donate they all choose one of 11 amounts. For each of these amounts I tested if they were over represented by dictators who saw one image compared to the other. For each amount I encoded the data as 1 if they donated this amount and 0 for any other amount and ran a fishers exact test for each amount (11 tests in total). The only one that was significant was $0.50, those that saw one image gave $0.50 significantly more often than those that saw the other.

I realize I have gone overboard with tests. I am not too sure too which I should apply an adjustment. ie. If I Bonferroni adjust for the kruscall-wallis test do I just divide alpha by 6 as this is how many levels in income and Education or do I need to take into account the Mann-Whitney tests and Spearmans rank. Also, for the Fishers exact tests the data is encoded differently for each test. Therefore do I need to make a correction or not? What sort of correction should I make?



1 Answer 1


You might want to refer to this answer to a related question.

Strictly speaking, alpha gets inflated with your extra tests on the one data set. However, correcting for alpha would increase type 2 errors.

You have to ask yourself, are any of these tests fishing expeditions? Does Type I or Type II error matter more? Are all my tests well theoretically motivated? Are any redundant? Those questions and more will help you assess whether you should correct for alpha.

But most important, you should be focusing on the effect size and credibility of your effects. Start thinking about your findings not in terms of a bunch of discrete decisions about significant and not, but as findings with a magnitude and range and meaning with respect to that magnitude. Your experience of data analysis, understanding of your work, and genuine value of your contribution to science will all improve.

That's general advice for thinking about your data. For your specific case, you should consider the distrbution of your data. I suspect you have a problematic ceiling effect not necessarily solved by your selection of non-parametric tests. You may have to treat that 0.5 donation special. You have to examine the data first and perhaps include that in your question.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.