I ran an ANOVA test on some data. The Omnibus F-test comes back with a P-value of 0.3726. I was taught that this means that none of the individual variables would show as significant. But one of my variables has a p-value of 0.0303. I am unsure how to proceed. Here is my sas output:

Source          DF  Sum of Squares Mean Square  F Value   Pr > F 
Model            8      673389798    84173725     1.12    0.3726 
Error           36     2702298024    75063834     
Corrected Total 44     3375687822       

 R-Square Coeff Var Root MSE Salary Mean 
 0.199482 22.85701  8663.939 37904.96 

Source               DF   Anova SS   Mean Square  F Value   Pr > F 
Experience            2 579574346.8  289787173.4    3.86    0.0303 
Education_Level       2  83814739.4   41907369.7    0.56    0.5771 
Experienc*Education_  4  10000711.3    2500177.8    0.03    0.9978 
  • $\begingroup$ I can't read the output due to its formatting, but if you have one predictor that makes a difference and a few others that really don't, then the MS for the model becomes diluted by the latter. Remember the numerator of the $F$ stat is divided by the number of predictors. $\endgroup$
    – Russ Lenth
    Apr 14 '15 at 21:30
  • $\begingroup$ PS if you indent each line of computer output by 4 spaces, it miraculously becomes readable. Can you make that edit? $\endgroup$
    – Russ Lenth
    Apr 14 '15 at 21:33
  • $\begingroup$ "I was taught that this means that none of the individual variables would show as significant" -- if that's the case, it seems possible you were mistaught. The situation you have can easily happen. $\endgroup$
    – Glen_b
    Apr 15 '15 at 2:00
  • $\begingroup$ ... However, if the omnibus test is not significant, it may be that you choose not to test for the individual variables. That's not to say none would be significant if you looked at them. $\endgroup$
    – Glen_b
    Apr 15 '15 at 3:45
  • $\begingroup$ You may find this post helpful:stats.stackexchange.com/questions/28845/… $\endgroup$ Apr 15 '15 at 21:25

It's generally good to think of ANOVA results in terms of the mathematical hypotheses they test. The omnibus test, tests whether all your means are equal, i.e. $$\mu_1=\mu_2=...=\mu_j$$ Your omnibus test (model, line 1 of output) is not significant, which means you do not have enough evidence to reject that hypothesis. Classically, people think that means you shouldn't move forward with more specific analyses. However, omnibus test can non-significant for multiple reasons. It could be that all your means are in fact equal. However, it could also be that a lot of your means are equal, and one or two isn't, but because you have a lot of means (i.e. a lot of groups), you have lost too many degrees of freedom to adequately test the omnibus hypothesis. For this reason many people ignore the omnibus test, and focus on the hypotheses they intended to test before the experiment began.

Your test of the experience factor tests if there are mean differences between the groups defined by levels of experience. Let's say groups 1-3 were in the same level of experience, 4-6 were the same level, and 7-9 were the same level. The hypothesis being tested is $$mean(\mu_1,\mu_2,\mu_3)=mean(\mu_4,\mu_5,\mu_6)=mean(\mu_7,\mu_8,\mu_9)$$ According to your output, this hypothesis can be rejected. There is some contrast between the 3 groups that reveals a statistically significant difference in means.

Your remaining tests (education and the interactions) test other hypotheses similar to the experience one. They are not significant. Because these are multiple degree of freedom tests, they are similar to the omnibus test. They say you don't have enough evidence to reject that means of the 3 groups defined by levels of education are all equal. However, if you created contrasts between the 3 education levels, one may be significant.

In short, multiple degree of freedom tests are intended to say whether differences in your data could have arisen from chance. However, there are multiple interpretations of their results. One interpretation says to stop where you are; the model is poor and should not be interpreted. Another says that you don't care about any significant differences; you care about a specific pattern of results that you hypothesized before the experiment. Thus, myself and most other people don't put too much weight into omnibus tests. If they are not significant, chances are your hypotheses aren't supported anyway, and the study isn't really publishable (or whatever your goal is). However, the evidence it does reveal can inform future research (e.g. from your results you have evidence that if there is an effect of education, it is likely smaller than the effect of experience, because you were able to detect the experience effect with a small sample size, but could not detect the education effect).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.