3
$\begingroup$

I want to compare two independent samples (from A/B test). T-test is most sensitive for normally distributed data (and will give smaller p-value than U-test if there is a real shift in mean value). But T-test performs poorly when there are large outliers: it would give high p-values. And here U-test is much better. So in practice one should look at data, decide how normal are the values and whether the outliers would be a problem. And then choose which test to use.

But if T-test would give smaller p-values for normally distributed data, while U-test for other. can I make a decision about how normal are the data by comparision p-values of T-test and U-test?

And finally I just calculate p-values from both tests and pick the smallest without looking at data. If they are normally distributed, than T-test would give the smaller p-value (which is good), in other case the U-test will give the smaller p-value (which is again good). So sounds like a universal approach. Is it a good idea? Why haven't I head that solution as a best practice for those who wants best results without digging into data?

I've tested this idea by generating random data with different distributions and for all examples I've made the combined test was correct. So if you state it is wrong approach please help with counterexample.

$\endgroup$
3
  • 4
    $\begingroup$ Why do you care about small p-values ? $\endgroup$ – Robert Long Jun 5 at 14:20
  • 4
    $\begingroup$ This question nicely illustrates the problem about deciding which test to use from graphical checks - if you can decide whether to use U-test or T-test from a graphical check, why can't you use other clues such as the p values of those tests. The answer is that you are technically not allowed to use either, because it skews the significance level of the test. However, if we followed that we wouldn't know which test to use, and so we have decided that graphical checks are okay anyway. $\endgroup$ – svendvn Jun 5 at 14:21
  • $\begingroup$ There are many dimensions even to this specific question, as the discussion already shows. Personally I would always prefer to persuade people that there is a better way to think about their question than to assert or insinuate that their procedure is illogical or dishonest. My bemusement starts at the idea that t test and U test are answers to the same question, which leads to focus on what is the real question. I don't think the P-value of a test depends on whichever test you did or did not also do. $\endgroup$ – Nick Cox Jun 7 at 10:03
7
$\begingroup$

A counter-example is to simulate two normally distributed samples with the same mean many times and see if at most 5% of the tests lead to rejection of the null hypothesis (if 5% is the desired significance level). 10,000 times, I simulated two samples of size 100 and ran your proposed procedure. Then I counted how many times it led to rejection and compared it with 5%. The result was

$$ 0.0601 [ 0.0556; 0.0649] $$

This means that the significance level can't be 5%, and hence it is not a valid statistical test for normally distributed data (or at least not one with a known significance level).

It should be possible to do some correction of the proposed procedure such that it is a valid test for normally distributed data, but I don't see how you could make a procedure like this valid for data for the class of distributions for which the U-test is valid. If I were you, I would therefore just use the U-test for this task.


R code:

one_test=function(){
  data1=rnorm(100)
  data2=rnorm(100)
  u1=t.test(data1, data2)$p.value
  u2=wilcox.test(data1,data2)$p.value
  p_value=min(u1,u2)
  return(p_value<=0.05)
}
prop.test(sum(replicate(10000,one_test())),10000,p=0.05)
$\endgroup$
1
  • $\begingroup$ Thank you for the specific example. It looks like the first answer which points out exactly to the problem. $\endgroup$ – Leonid Mednikov Jun 5 at 16:13
7
$\begingroup$

There is no such thing as a "best result without digging into data." What you are describing is a very bad statistical taboo - null hypothesis testing is fraught to begin with, and this sort of procedure is one of the easiest ways to render it totally useless.

I'd suggest you strongly reconsider your intuition that a small p-value is "good." A p-value means very little without additional context, and any sort of procedure that explicitly searches for small p-values without looking at the data is at best useless, and at worst dishonest.

Edit: As suggested below, those who wish to learn more about this issue can read the relevant wikipedia article.

$\endgroup$
3
  • $\begingroup$ Can you describe at least one example of the problem I can encounter? And the reason I don't want to look at data is that I'm thinking of A/B test in large project where we have 100+ measured variables and 10+ A/B test at one time. So yes, I don't want to look at 1000+ distribution and make decision each time. So currently we just use U-test as it is more universal. And my question, why is may make a problem to mix it with T-test, so for normally distributed variables will have better sensitivity. $\endgroup$ – Leonid Mednikov Jun 5 at 13:37
  • 1
    $\begingroup$ The "problem you can encounter" is "significant" results that do not correspond to anything meaningful. $\endgroup$ – user3716267 Jun 5 at 14:30
  • 1
    $\begingroup$ Good answer. Maybe you can give a link to en.wikipedia.org/wiki/Data_dredging, which discusses the problem. $\endgroup$ – John Coleman Jun 6 at 11:49
4
$\begingroup$

You should decide your test first, then get the p-value. Otherwise you could keep doing tests until one of them gives you what you want. If you're going to perform multiple tests, then you should use a correction on the significance level (like Bonferroni's)

$\endgroup$
5
  • 1
    $\begingroup$ I don't find comparision with Bonferroni correction relevant. Let's say I've set a p-value before looking at distribution. Then I look at it and make decision to calculate T-test or U-test. Is this procedure correct? If yes, than my question: isn't it right, that for normal distribution T-test will give smaller value, for other distributions U-test will give smaller value. So instead of deciding which is the distribution I'll just calc both p-values and get the minimum. $\endgroup$ – Leonid Mednikov Jun 5 at 13:44
  • 1
    $\begingroup$ @LeonidMednikov why not do a million different test for a million different possible distribution and pick the lowest of those? $\endgroup$ – David Jun 5 at 14:41
  • 1
    $\begingroup$ @LeonidMednikov just to be sure you know; the T-test will not always give a smaller p-value if the data is normally distributed - on average, yes it will be lower, but it is those times where it is not that messes up the significance level of your proposed procedure (and vice-versa for U-test and non-normal data). $\endgroup$ – svendvn Jun 5 at 14:57
  • $\begingroup$ @David, I think the answer is in comment under the question. I would rephrase like this: why is it allowed to perform a test for normality to choose a T/U-test, but my procedure isn't allowed? But actually the answer seems to be at the next after yours comment here. My procedure is correct on average, but in particular cases U-test might give smaller p-value than T-test even on normally distributed data. $\endgroup$ – Leonid Mednikov Jun 5 at 16:28
  • $\begingroup$ @svendvn Thank you. That is the clue of my mistake: I saw that T-test gave lower p-value on average for normal distribution, and assumed that it is lower for every case, which is wrong. $\endgroup$ – Leonid Mednikov Jun 5 at 16:31
-1
$\begingroup$

To sum up all the topic

When two samples are from the the same normal distribution, T-test and U-test would give almost the same p-values, sometimes T-test would be smaller, sometimes U-test. So combined min(T-test, U-test) will give lower p-values that each of them and we'll have more rejections of null hypothesis than expected. That is the problem.

Taking into account that even for normal distribution the benefits of T-test are quite small, if you don't check distribution and outliers just use U-test.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.