4
$\begingroup$

I undertook a large study (N about 200 in both control and treatment) in which one of the user ratings is significantly different (p < 0.0001). When I ran the unpaired t-test, the F-test also returned significant (p = 0.0003). This violates the t-test assumption of equal variances, but I figured this was OK with these large sample sizes (it's still highly significant with Welch's correction anyway). The t-test tells me that the two means are different, and the F-test tells me the standard deviations are different. Should and can I report both? Whenever I see a site talking about reporting the F-test, it's always in the context of ANOVA, never alongside a t-test. This makes me think that reporting it here isn't common practice.

$\endgroup$
  • 4
    $\begingroup$ Why did you run both tests? What questions you were looking answers for? You use those methods to test certain hypotheses and report the results that are consistent with your research questions. There is not point in running any possible test... $\endgroup$ – Tim Dec 26 '14 at 16:28
  • $\begingroup$ @Tim the F test was for equality of variances and the t test was for equality of means, that's why. $\endgroup$ – wcampbell Dec 26 '14 at 16:36
  • 4
    $\begingroup$ Yes, I know. But why did you use them? What were you interested in? If you want an answer for both questions then you report both. $\endgroup$ – Tim Dec 26 '14 at 16:41
  • 1
    $\begingroup$ "Whenever I see a site talking about reporting the F-test, it's always in the context of ANOVA, never alongside a t-test" - but is that because you're looking at a different type of F-test? In particular, for linear models such as ANOVA or regression, an F-test is to see if the overall model is significant (compared to a null model) whereas you seem to be using an F-test to compare variances? $\endgroup$ – Silverfish Dec 26 '14 at 23:33
7
$\begingroup$

It's not clear what you did your F-test for, but I am assuming for now it was a test for equality of variances. If not I will edit my answer. But if it was a test for equality of variances, it probably wasn't a good choice because the F-test assumes that your data were drawn from a normally-distribution population, and is very sensitive to departures from normality. There are alternative tests which are more robust - see this question for discussion. But before choosing a better test for equality of variances, you should consider whether you really want to test for it at all.

A two-stage approach of using an equality of variances test before deciding on whether to use a t-test or non-parametric test, is strongly advised against because it means the operating characteristics of your test are uncertain - see also this question which is closely related to yours. For a reference, see Zimmerman, D.W. (2004), "A note on preliminary tests of equality of variances", Br. J. Math. Stat. Psychol., May; 57(Pt 1): 173-81.

It's a good idea to use a Welch-corrected t-test if you're in any doubt whether the equality of variances will apply - see this question and accompanying citations. If the variances are similar in your samples, you'll only have a small degrees of freedom penalty applied, so you don't need to worry that you'll lose lots of power by applying the Welch correction in a case where the variances are equal.

If you don't even think that the conditions for the Welch t-test are met, then you should be applying a different test such as Wilcoxon-Mann-Whitney. It need not be a big worry that, should conditions for a traditional t-test hold, that you have needlessly lost power, since asymptotically it will have $\frac{3}{\pi} \approx 95\%$ of the power (in large samples) anyway. In small samples simulations show rank tests don't perform much worse than t-tests either. Another alternative might be to apply a Welch t-test on the ranks of your data: it's been suggested this approach is particularly effective when variances are unequal, see Zimmerman DW and Zumbo BN, (1993), "Rank transformations and the power of the Student t-test and Welch t′-test for non-normal populations", Canadian Journal Experimental Psychology, 47: 523–39.

Is the (in)equality of variances something you are generally interested in, in its own right? If so, then it seems perfectly sensible to state whether the difference in variances was statistically significant by writing up the test results (but preferably not an F-test!). If this is simply a side-issue to deciding which test to apply in the first place, then you needn't bother.

$\endgroup$
4
$\begingroup$

It's not clear why you did both.

Note that the F-test for variances is very sensitive to the assumption of normality; there are better choices for a test of that, if you actually need a formal test.

If you did the test of variances as a check on the assumptions, it's not advisable to test that and choose a procedure based on its outcome. Better to simply not assume constant variance (the Welch test is a good default if you can't reasonably assume equal variance from the outset). [In that case, I'd simply ignore everything you've done as if you hadn't done it and instead do a Welch test and report that.]

If you're actually interested in whether there's a difference in either mean or standard deviation of treatments, then it makes sense to test for (and report) both -- but I still wouldn't do the F-test for variances in that case.

$\endgroup$
  • $\begingroup$ (+1) Good point re F-test. $\endgroup$ – Silverfish Dec 27 '14 at 1:29
0
$\begingroup$

Another approach would be to use the results from the F-test to justify using a test that does not assume equality of means. You could use a nonparametric approach such as the Wilcoxon rank-sum test instead of the $t$ test.

The nonparametric alternatives are more powerful than the $t$ test (and thus preferred) when assumptions are violated. You have plenty of data to run these and I would expect that you will get the same result as with the $t$ test.

$\endgroup$
  • $\begingroup$ "Another approach would be to use the results from the F-test to justify using a test that does not assume equality of means" - I don't think this is wise. A major problem with using two stages like this - a first test to decide which test to use in the second test - is that the operating characteristics become uncertain. See this question for some extensive discussion $\endgroup$ – Silverfish Dec 26 '14 at 23:47
0
$\begingroup$

Your question is confusing because you haven't described the experimental design or the tests you used in enough detail for a reader to be sure what you did. What I assume you mean is that you ran Levene's test for equality of variances and found a significant result?

In that case, if you are running a one-way ANOVA, yes, you should run Welch's test and report the F value, p value, and degrees of freedom in the style recommended by whichever group governs the journals you publish in.

The Wilcoxon-Mann-Whitney is also an option, though Welch's is usually more powerful (see Tomarken and Serlin 1986.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.