2
$\begingroup$

We have a data-set with two variables - gender and education qualification. The data-set is severely unbalanced. The are only ~10 observations from 1 level of the education factor, ~20 from another and ~80 from another. Moreover the sampling of gender is not even within each level of education.

We run an ANOVA in SPSS with Type III SS, there is no significant interaction, and education is significant while gender is not. However, I have read that when datasets are very unbalanced we should be careful that Type III analyses do not overlook effects.

Sure enough, when we run ANOVA with type I SS with gender entered first in the model, it comes out as strongly significant. When education is entered first in the model, it is strongly significant.

What are our options here. Given the absence of an interaction, can we just carry out the ANOVA w/TYpe I SS twice with the re-ordered factors and report the results of each?

$\endgroup$
  • $\begingroup$ Schemes like these can help a bit to understand consequences of unbalanced designs. I surmise your case is unbalanced disproportional? You probably are adviced to use type III. Yes you'll lose (overlook) some power of effects, but this type is the only one which tests exactly the same hypotheses in unbalanced case as in balanced case. Note also that if the unbalanced case is proportional and interaction effect is missing in the model, type II = type III. $\endgroup$ – ttnphns Jan 14 '17 at 22:47
  • $\begingroup$ And I doubt that you will be considering type I - "hierarchical" model, with your specific two factors. Why do it, what theory might lead to it? $\endgroup$ – ttnphns Jan 14 '17 at 22:51
  • $\begingroup$ Thanks "ttnphns". The full data-set does have additional explanatory variables (6 other factors i think) - sorry this wasnt mentioned initially - though the researchers are mainly interested in gender and education. Is it feasible to use SS type I to analyse the model, given the focus on the main effects of certain variables + the fact that the inherent confounding of certain variables will potentially lead to overlooking of significant main effects when using type III SS? If we were to use Type I SS, how does one justify the ordering of terms in a complex model with numerous terms? $\endgroup$ – RBW Jan 15 '17 at 0:11
  • $\begingroup$ It's not clear to me that there is a consensus. $\endgroup$ – Glen_b -Reinstate Monica Jan 15 '17 at 2:07
1
$\begingroup$

Because of the unbalanced design, gender and qualification are confounded. The safest thing to say is that gender, qualification, or both have effects but, because of the confounding, there is probably no justifiable way to separate the effects. If you carry out the Type I sums of squares tests twice, each variable that is entered first will be confounded with the one entered second.

$\endgroup$
  • $\begingroup$ Thanks David. The full data-set does have additional explanatory variables (6 other factors i think) - sorry this wasnt mentioned initially - though the researchers are mainly interested in gender and education. Is it feasible to use SS type I to analyse the model, given the focus on the main effects of certain variables + the fact that the inherent confounding of certain variables will potentially lead to overlooking of significant main effects when using type III SS? If we were to use Type I SS, how does one justify the ordering of terms in a complex model with numerous terms? $\endgroup$ – RBW Jan 15 '17 at 0:12
  • $\begingroup$ I like to use somewhat different language than the standard because I think it is more straightforward. Instead of "ordering of terms" I would say "allocation of confounded variance." If you are going to allocate confounded variance to one of your independent variables then I think it would be a good idea to justify it. Sometimes this can be done based on the direction of causality but it can be difficult. One approach is to show the effects of different ways of allocating confounded variance. Ultimately, though, the uncertainty about the main effects should be acknowledged. $\endgroup$ – David Lane Jan 15 '17 at 0:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.