I got very confused while looking at help examples of the t.test function

t.test(1:10, y = c(7:20))      # P = .00001855

t.test(1:10, y = c(7:20, 200)) # P = .1245    -- NOT significant anymore

Why should adding the value 200 increase the p-value? Shouldn't we get a smaller p-value and thus rejecting the null hypothesis? Notice that a second element would cause the p-value to drop again.

t.test(1:10, y = c(7:20, 200,200))  # P = 0.06883
  • 5
    $\begingroup$ t.test uses variance estimate. By adding 200 you increase the variance of the second sample significantly, so the "confidence" of the test drops. $\endgroup$
    – tonytonov
    Nov 7 '14 at 14:53
  • 1
    $\begingroup$ The t-test assumes normality (or at least a close approximation). Your second example contains an outlier and the sample size is small. IIRC Welch's t-test is particularly sensitive against this. $\endgroup$
    – Roland
    Nov 7 '14 at 14:54
  • $\begingroup$ P value remains less than 0.05 till t.test(1:10, y = c(7:20, 115)). It progressively increases as the added value increases due to increasing variance. $\endgroup$
    – rnso
    Nov 7 '14 at 15:29
  • $\begingroup$ I see that your blaming the increase of the variance, but for me this is a weird behaviour of failing to reject the null hypothesis instead of doing so. Is there any other test which avoids such a thing? $\endgroup$
    – Shaki
    Nov 7 '14 at 16:02
  • $\begingroup$ This is the opposite of "weird behavior": the outlier should cause us to be extremely cautious about the result, so it is good that the p-value increases. With this outlier present it is no longer at all evident that the means of the two populations differ, which is what the t-test is assessing. $\endgroup$
    – whuber
    Nov 7 '14 at 16:37

The standard deviation that comes into the denominator also affected by the outlying observation. (It also affects the degrees of freedom because we are actually doing a Welch test here.)

We can examine the influence of changes in that additional observation on the t-statistic:

tinfl <- function(x) t.test(1:10, y = c(7:20, x))$statistic

(or the p-value tinflp <- function(x) t.test(1:10, y = c(7:20, x))$p.value).

[Adding an observation near 10.18855 leaves the test statistic about where it was without the additional observation (an observation near 9.666492 leaves the p-value where it was).]

This is called an empirical influence function. It's useful for seeing how statistics respond to moving a data point.

So here's what happens to the numerator, denominator and value of the t-statistic as we vary that additional observation:

enter image description here

The red dashed line marks what happens when the additional observation is 10 (somewhere close to where it needs to be to get either the same test statistic or the same p-value as without the observation).

As you move the additional observation up from that, the t-statistic becomes more and more negative (more significant) until you hit about -17.25, and then the effect on the standard deviation (and to some extent the d.f.) starts pulling it back.

As $x\to\infty$, the test statistic goes to -1.

(the limit as $x\to -\infty$ is 1)

You see similar effects with the equal-variance two-sample t-test as well.

The t-test is not particularly robust to very large outliers.

If you had two processes you were interested in identifying location differences in, but there was rare contamination by extreme outliers (from some additional process which was not of interest in the things you wanted to compare with the test), you could robustify the t-test (by modifying the influence function of both numerator and denominator so that they're both bounded), or consider say a Wilcoxon-Mann-Whitney test, so that the effect of the extra observation is more like what you might expect. Or you might consider a permutation test (whether with a robust statistic or not).

Here's the effect on the Wilcoxon-Mann-Whitney test, and a particular form of robustified t-test for comparison - as you see, the Wilcoxon statistic is monotonic, while the robustified t-test only comes back slightly.

$\hspace{1.5cm}$enter image description here

  • $\begingroup$ The W-M-W test has the same inferential properties as the K_S test (which was developed earlier). Remembering it as the Kolmogorov-Smirnow test has the advantage for R users that the stats-function is ks.test. $\endgroup$
    – DWin
    Nov 8 '14 at 4:45
  • $\begingroup$ @DWin When you say "has the same inferential properties" I'm not quite clear what you intend to convey. Can you be more explicit? $\endgroup$
    – Glen_b
    Nov 8 '14 at 5:17
  • $\begingroup$ It will yield the same p-values for the same data. $\endgroup$
    – DWin
    Nov 8 '14 at 5:55
  • $\begingroup$ @DWin This is not the case in general. You may be confusing the test with some other test. $\endgroup$
    – Glen_b
    Nov 8 '14 at 6:34
  • 2
    $\begingroup$ @DWin Yes, that's correct, the statistics are at worst linear functions of each other -- which is why I refer to it as the Wilcoxon-Mann-Whitney $\endgroup$
    – Glen_b
    Nov 8 '14 at 6:41

The two-group t-test balances (by computing their ratios) two aspects of the distributions of the values, one: the difference in the means, which your "disturbance" did increase somewhat, and two: the variance which your disturbance increases much more. The variance varies with the square of the differences of the point values from the mean and your single extra value that was hundreds of units from the mean would have added something on the order of 180^2 to the variance estimate where it had been much smaller. Under the hypothesis of normally distributed values (which your alteration has massively violated), the variance is now much larger and the estimate of the difference in group means now plausibly includes zero.

If you want to use a test statistic that does not depend on squared deviations, consider the ks.test:

 ks.test(1:10, y = c(7:20, 200))
    Two-sample Kolmogorov-Smirnov test

data:  1:10 and c(7:20, 200)
D = 0.7333, p-value = 0.003151
alternative hypothesis: two-sided

Warning message:
In ks.test(1:10, y = c(7:20, 200)) : cannot compute exact p-value with ties

As became clear in my correspondence with Glen_b I was conflating the wilcox.test in R with the ks.test in R. I meant to use the the wilcox.test (Wilcoxon Rank Sum test):

> wilcox.test(1:10, y = c(7:20, 200))

    Wilcoxon rank sum test with continuity correction

data:  1:10 and c(7:20, 200)
W = 8, p-value = 0.0002229
alternative hypothesis: true location shift is not equal to 0

Warning message:
In wilcox.test.default(1:10, y = c(7:20, 200)) :
  cannot compute exact p-value with ties

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.