# Validity of t-test, Welch's t-test and Mann-Whitney U test

Suppose you wanted to compare whether the returns of stocks A and B had the same mean. It is unlikely they have the same variance.

To do this you performed a t-test, Welch's t-test and Mann-Whitney U test on some data.

Acknowledging the fact that stock returns are non-normal (e.g., they are leptokurtic) and dependent, to what extent, if at all, are these hypothesis tests valid? Are they at all useful?

• In what sense are your data dependent? A lot depends on that. – gung - Reinstate Monica Nov 12 '18 at 21:28

Yes, I think all three mentioned tests will give you valid information regarding the difference in average stock returns.

The t-test is already pretty good because it relies on the t distribution that is leptokurtic with fatter tails (similar to your stock returns distribution). The Welch's test is essentially a t-test accommodating for two samples of different size and with different variance. The Mann-Whitney test is even more distribution independent than the first two because it gets away from nominal value and deals instead with the values' rank.

In such circumstances, statisticians will often suggest that those tests in the order mentioned are progressively better for the mentioned reasons (Welch's better than t, and MW better than Welch). However, in reality you will most often observe that it makes very little difference. A few years back, I used such tests extensively in numerous different situations. And, invariably all three tests gave me directionally very similar results. I can't recall a situation where the tests generated divergent results.

To check if your average returns are different, sometimes just using a graph and smoothing it can be very informative. The graph will provide you a lot of visual information way beyond just eyeballing average returns. You will observe the volatility of the two stock returns and the correlation between the two. As fleshed out in the next paragraph, those considerations are as important if not more than just considering "averages."

Keep in mind that comparing two stocks returns by their average only touches on the tip of the iceberg. Their respective volatility (captured by their standard deviation) is just as important if not more than their average return. The Sharpe ratio factors that in and provides a measure of returns generated for the risk taken. Also, the correlation of your two stocks is extremely important. Even if stock A has a higher risk and lower return than stock B, stock A may provide diversification benefit if its correlation with stock B is low or negative. Granted that is unlikely. Few stock returns have a negative correlation to each other. But, that is something to look at. If you do [have negative correlation between the two] that is a gem worth uncovering and not throwing away. As you know, truer portfolio diversification is obtained by investing in different asset classes, not just stocks.

• The null tested by MW is different from the null tested by t-tests. The Welch t-test is for differing variances, not n's; it's just as valid if the n's are the same, & it's just as unnecessary if the n's differ but the variances are the same. Finally, the fat-tails of the t-test do not provide robustness to a fat-tailed distribution of the data--that's just a misconception. – gung - Reinstate Monica Nov 12 '18 at 21:28
• I beg to differ on several counts. Welch is specifically earmarked for both unequal variance and unequal sample if necessary. en.wikipedia.org/wiki/Welch%27s_t-test. The t-distribution is leptokurtic just as the one of stock returns. Often stock returns are not that far off a Normal distribution anyway, especially if factoring for fatter tails as in the t distribution. The MW null is that when the data are combined together, each sample has the same average rank... which is a nonparametric correspondence to comparing nominal average values. – Sympa Nov 13 '18 at 1:27
• The Wikipedia article is largely, but not perfectly correct. Type I errors are most inflated if a larger SD is combined with a smaller n, but Welch's t is appropriate if variances differ even if the n's are the same, & certainly isn't necessary if n's differ but variances don't. Simple simulations show this. The t-distribution pertains to the distribution of the t-statistic; the raw data are assumed to come from a normal population, see Wikipedia if you like. – gung - Reinstate Monica Nov 13 '18 at 2:21
• Lastly, the average rank is not the same as average value. MW tests for stochastic dominance, not equality of means, see: How can a Mann-Whitney U-Test return a p = 1.00 for unequal means? & Wilcoxon-Mann-Whitney test giving surprising results. – gung - Reinstate Monica Nov 13 '18 at 2:25

At the time of writing, the question is not clear in what sense the data are dependent. I might guess that there is an assumption that two stocks are correlated by date. That is, if on a given date the stock market is generally high, that both stocks might be relatively high as well.

If this is the case, then a paired test might be applicable. Appropriate tests might include paired t-test, (paired) Wilcoxon signed-rank test, and sign test. For each of these tests, it would be the difference between the stocks on each date that would be the quantity of interest.

However, I think it's necessary to describe what is meant with "return". How this is defined or calculated will likely have an effect on what analysis would be appropriate.