I a have two independent samples one with more than 700 observations and the second with more than 500 observations. I firstly use Cramér-von Mises test to test normality just to check how non-normal the distribution of the datasets is. It returned me p-value equal to 0. Thus, I have to rely on CLT (hope that I have enough observations - or should I rather somehow "test" that?). I then used Conover test to test equality of variances and it again returned me p-value equal to almost zero. Therefore, I want to compute the unequal variance t-test (Welch test), but I am wondering whether it was better to used standard Student's t-test because as I understand that due to the Lindenberg CLT, we can disregard the unequality of variances. So, generally, which test should I use? And was the above-mentioned testing of data reasonable?

  • 2
    $\begingroup$ A goodness of fit test (C-vM or otherwise) doesn't tell you how non-normal a distribution is -- a p-value is not an effect size. Further, how much impact a given amount of non-normality has on a t-test decreases with sample size, while the probability you'll reject normality at a given amount of non-normality increases with sample size ... so you become most likely to reject just when it matters least. Further, choosing a test on the basis of a formal test of assumptions affects the properties of the second test (sometimes badly); a number of authors recommend against the practice. $\endgroup$
    – Glen_b
    Apr 18 '15 at 6:06
  • $\begingroup$ Thanks, @Glen_b. I heard that it is better to straightly use Welch test unless we have strong reasons that the variances are equal. So is that what you would recommend? And do you think that my sample sizes are large enough to rely on CLT even in case of "strongly" non-normal distributions (I know that C-vM did not tell me that it is "strongly" non-normal). And is it true that due to the Lindenberg CLT, it is meaningless to test variances? $\endgroup$ Apr 18 '15 at 6:18
  • $\begingroup$ See this answer and this answer and my first comment here and more broadly, ... (ctd) $\endgroup$
    – Glen_b
    Apr 18 '15 at 6:41
  • $\begingroup$ (ctd)... this (there are many more). I can't guess whether your data are such that you can reasonably treat them as normal or not, since I have no basis for considering the type nor extent of the non-normality at hand -- but for moderate amounts of skewness or heavy-tailedness or discreteness you should be just fine. It could be much more skew than exponential without problems but it's easy to find cases where a much larger n isn't enough. $\endgroup$
    – Glen_b
    Apr 18 '15 at 6:44
  • $\begingroup$ Thanks, @Glen_b. I went through all the texts you posted and my conclusion from that is: I will not check for normality (maybe only at the end of the work, I will just provide the p-value of C-vM test) and assume that CLT will suffice for normality (1st sample: skew=4.6, kurt=34; 2nd sample: skew=2.2, kurt=13.2). Then I will directly use Welch test (argumenting that it is probable that variances are unequal if only for the reason that sample sizes are not equal) and also disregarding the Lindenberg CLT (for un/equality of variances). Is that a good approach? $\endgroup$ Apr 18 '15 at 8:27