Parametric and non-parametric tests?

Question

I am trying to do tests to see if two numbers were derived from similar distributions. I am not a statistician so please excuse my use of words. I will try to be as accurate as possible. I have a two sets of numbers. If i do a Student's T-Test (unpaired, two way), i get a p-value that is much greater than 0.05. However, for the same sets of numbers if i do Mann-Whitney U test, or the Kolmogorov-Smirnoff test, the p-value is much smaller than 0.05.

Is such a behavior expected. And should i consider the differences a significant or not.

EDIT: After reading the answer posted, I guess I have to add one more subquestion, What do these tests test?

Answer 1

This behavior is certainly possible, but there are some other things that you should check as well.

If the p-values are like 0.99 vs. 0.01 then it is possible that you are testing the wrong tail and the tests really do agree. Also check that you are doing all the computations correctly and using the exact same data.

One case where what you describe is likely is when there is an outlier or 2, the t-test is testing for the difference in the means and the power of the test is related to how different the means are relative to the total variability. An outlier or 2 can greatly increase that variability and therefore not have the power to distinguish between the means. The other tests are not affected by the outliers (but also test a different null hypothesis) so can find the differences.

Since each of the tests are testing a different null hypothesis it is possible to get different answers and have all the answers be correct. Consider 2 populations, the first has 90% of the values equal to 1 and 10% equal to -9, the other has 90% of the values equal to -1 and the remaining 10% equal to 9. Both of these populations have a mean of 0, so any test comparing means will give the correct decision when the null is not rejected. However, the medians are different, so a test on the medians (MW is not a test on the medians) should reject the null. The MW test null is that when you look at all pairwise comparisons between the 2 populations that each populations' values will be greater than the other 50% of the time. In this example 81% of the time the comparison will be 1 vs. -1, 9% of the time 1 vs. 9, 9% of the time -9 vs. -1, and 1% of the time -9 vs 9; so the correct decision here is to reject the null (the proportion is 81% not 50%). The KS test looks at the shape and these shapes are very different, so it should reject as well (though the standard KS test is not well suited for the ties that this would produce).

The important thing is to think about what you really want to test and look for a test that will actually test that and that has assumptions you can live with based on the science behind the data.