2
$\begingroup$

I have a very small data set, with two groups, 5 cases in each one. I ran the t-test, and got a p value of 0.049. The F test that SAS ran in addition gave a p value of 0.72. This is not a surprise, even if the variances differ, they need to differ "a lot" for the test to be significant with 5 cases per group. So, I tried ignoring the F test and looked at the t-test with Satterthwaite approximation of the degrees of freedom, and guess what ? The p value is 0.051. What should be my conclusion ? How to I decide in such a case ? The mean of group 1 is 34 and of group 2 is 12. The standard deviations are 16.4 and 13.5 respectively. By the way, I tried running it again in Stata, and it allows me to ask for the Welch approximation of degrees of freedom, the p value in this case was 0.043, but I read in an old post here that the Satterthwaite is more common in use.

$\endgroup$
1
$\begingroup$

You have equal sample sizes - the t-test is quite insensitive to the equal-variance assumption in that case.

(The standard deviations are also very close; the equal variance t-test is not that sensitive to mildly unequal variances even when the samples sizes aren't so close to equal.)

There should be no problem with using either test. The two tests should tell you about the same thing ... and in fact they do.

So there's no problem with using the equal variance t-test.

The only problem with what you did is not making the choice of which test you were using before seeing the results. That is problematic - if you are choosing between tests post-hoc your test procedures certainly won't have their nominal properties.

$\endgroup$
  • $\begingroup$ Thank you for the helpful comment. when you run the test on SAS (or SPSS for that matter), you get both tests, and the F test. So if you want to decide according to the F test, you will see the results before you decide. I could easily said that according to the F test I choose the equal variance test. I didn't do it, because I know that the F test would most likely won't be significant due to the sample size. $\endgroup$ – user70606 Mar 14 '15 at 10:13
  • $\begingroup$ What I am saying is that I have no trust and no belief in the mechanism of hypothesis testing. It makes no sense that under H0, the probability of randomly getting the test statistic (or an extremer value) is 4.9% - and this is a very low chance, so I reject H0, but 5.1%, wow, this is much higher, I can't reject H0. Is it only me or something with this mechanism just doesn't make sense ? $\endgroup$ – user70606 Mar 14 '15 at 10:13
  • 1
    $\begingroup$ It's really silly to make such hard-line accept-reject decisions. Just report the p value. .051 is not much different from .049. $\endgroup$ – rvl Mar 14 '15 at 19:24
  • $\begingroup$ @user70606: "I have no trust and no belief in the mechanism of hypothesis testing". I can sympathize. But in that case, don't use t-tests or F-tests at all. That's what they're for. $\endgroup$ – Kodiologist Mar 14 '15 at 22:08
  • $\begingroup$ what should I use then ? there is no alternative... $\endgroup$ – user70606 Mar 16 '15 at 6:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.