I want to validate a questionnaire with three scales summing up to a total score. For for all tests I conducted, I tested the scales separately and additionally the total score (so 3+1 and that total one bearing all the summed up information of the first 3 scales). I've tested:
- Skewness and Kurtosis of the scales to find out about deviation of normal distribution and Cronbach's Alpha for all scales
- An ANOVA to test gender differences
- independent samples t-tests for each scale (3+1) to test for group differences between psychology students and student of other fields of study
- Several product-moment-correlations with the scores of three further questionnaires measuring convergent or similar concepts and also sometimes having more that one scale (4/2/1)
So, I have a lot of p-values in the scope of the study.
What I don't understand:
- Do I really have to divide the Alpha level by the number of all the tests conducted (4*Skewness + 4*Kurtosis + 4*Alpha ( + even the alphas of all the other scales --> 4+2+1) + ANOVA (for all scales 4+4+2+1) plus 4*t-tests + (4*4 + 4*2 + 4*1) Product-moment-correlations. Or do I calculate one Alpha level for each scale in dependence of how often this scale was involved in any testing?
- Then I don't now how to handle the fact, that the fourth scale (the total score) consists of all the items of the three "subscales".