9
$\begingroup$

Here are four different sets of numbers:

A = {95.47, 87.90, 99.00}
B = {79.2, 75.3, 66.3}
C = {38.4, 40.4, 32.8}
D = {1.8, 1.2, 1.1}

Using a two-sample t-test without assuming equal variances, I compare B, C, and D to A and get the following p-values:

0.015827 (A vs B)
0.000283 (A vs C)
0.001190 (A vs D)

I find it strange that the p-value from the A-D test is worse than the A-C test: the difference between the means is clearly much bigger AND the variance of D is much lower than the variance of C. Intuitively (at least for my intuition), both these facts should drive the p-value lower.

Could someone explain if this is a desired or expected behavior of the t-test or whether it has to do more with my particular data set (extreme low sample size perhaps?). Is the t-test inappropriate for this particular set of data?

From a purely computational point of view, the reason for a worse p-value seems to be the degrees of freedom, which in the A-D comparison is 2.018 while it is 3.566 in the A-C comparison. But surely, if you just saw those numbers, wouldn't you think that there is stronger evidence for rejecting the null hypothesis in the A-D case compared to A-C?

Some might suggest that this is not a problem here since all p-values are quite low anyway. My problem is that these 3 tests are part of a suite of tests that I am performing. After correcting for multiple testing, the A-D comparison doesn't make the cut, while the A-C comparison does. Imagine plotting those numbers (say bar-plots with error bars as biologists often do) and trying to justify why C is significantly different from A but D is not... well, I can't.

Update: why this is really important

Let me clarify why this observation could have a great impact on interpreting past studies. In bioinfomatics, I have seen the t-test be applied to small sample sizes on a large scale (think differential gene expression of hundreds or thousands of genes, or the effect of many different drugs on a cell line, using only 3-5 replicates). The usual procedure is to do many t-tests (one for each gene or drug) followed by multiple testing correction, usually FDR. Given the above observation of the behaviour of Welch's t-test, this means that some of the very best cases are being systematically filtered out. Although most people will look at the actual data for the comparisons at the top of their list (the ones with best p-values), I don't know of anyone who will look through the list of all comparisons where the null hypothesis wasn't rejected.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Remember, the Welch formula is an approximation. Simulation studies indicate the "Welch correction becomes too conservative when sample sizes are strongly unequal," which is the case with the A-D comparison. $\endgroup$
    – whuber ♦
    May 9, 2012 at 21:28
  • 1
    $\begingroup$ The sample sizes are equal in this case @whuber. Did you mean sample variances? $\endgroup$
    – ALiX
    May 9, 2012 at 22:18
  • 1
    $\begingroup$ Thanks, ALiX, you're right. For the case of highly unequal variances and equal sample sizes, I should have quoted a different conclusion (which is even worse!): "... type I error ... becomes inflated to various degrees, so that the tests are invalid and should not be used." $\endgroup$
    – whuber ♦
    May 10, 2012 at 14:44
  • $\begingroup$ I wonder if your data are read counts from rna-seq? If so, may I suggest that you look into DESeq (R/Bioconductor package)? genomebiology.com/2010/11/10/R106 $\endgroup$
    – bdemarest
    May 12, 2012 at 3:02

4 Answers 4

Reset to default
3
$\begingroup$

Yes, it's the degrees of freedom. The t-statistics themselves increase as we compare groups B,C,D to A; the numerators get bigger and the denominators get smaller.

Why doesn't your approach "work"? Well, the Satterthwaite approximation for the degrees of freedom, and the reference distribution is (as the name suggests!) just an approximation. It would work fine if you had more samples in each group, and not hugely heavy-tailed data; 3 observations per group is really very small for most purposes. (Also, while p-values are useful for doing tests, they don't measure evidence and don't estimate parameters with direct interpretations in terms of data.)

If you really want to work out the exact distribution of the test statistic - and a better calibrated p-value - there are methods cited here that could be used. However, they rely on assuming Normality, an assumption you have no appreciable ability to check, here.

Improve this answer
$\endgroup$
6
  • $\begingroup$ More samples alone would not help: I would get lower p-values, but the order of the p-values would be the same. See my update why this could be important in some applications. $\endgroup$
    – ALiX
    May 9, 2012 at 20:17
  • $\begingroup$ I agree this phenomenon could be important, but more samples per group would help; asymptotically, the test gives accurate p-values. However, with tiny groups there are exact methods (e.g. SAM, by Tibshirani) which get their statistical validity from permutation-based analysis. $\endgroup$
    – guest
    May 10, 2012 at 0:51
  • $\begingroup$ If there is a large difference between the variances of the two groups, increased sample size won't help in the sense that the order of the p-values will be wrong (i.e., A-C will still have lower p-value than A-D). SAM cannot be used if you are testing the effect of many compounds at different concentrations (this should be an ideal application of the t-test). Welch's t-test appears to me to be fundamentally flawed: its purpose is to handle unequal variances, but the more unequal the variances are, the worse it performs (the degree of freedom approximation breaks down). $\endgroup$
    – ALiX
    May 10, 2012 at 3:52
  • 1
    $\begingroup$ If you want pairwise comparisons of the different concentration levels, and are testing gene expression levels over many genes, then SAM can be used for each pairwise comparison, and will give you honest statements of statistical significance for each comparison. You may then, should you wish, use these to rank the comparisons. Also, Welch's test is not fundamentally flawed. Sure, it just doesn't work well with n=3, but this is not what it claims to do. A bicycle pump is hopeless for peeling potatoes, but this doesn't mean you can conclude it is "fundamentally flawed". $\endgroup$
    – guest
    May 10, 2012 at 22:33
  • 1
    $\begingroup$ The order is potentially wrong no matter what you do, so that doesn't help. If you want to order by statistical significance of the between-group mean difference in your actual data, and have moderate size samples, the p-value from Welch's test will do fine. With tiny sample sizes, no, it won't do fine, but given that it was developed as an approximation and that it works better than many competitors, this is not cause for criticism. $\endgroup$
    – guest
    May 12, 2012 at 14:22
1
$\begingroup$

There is quite a bit to this question, and I'm fairly sure that some of it is beyond my understanding. Thus while I have a likely solution to the 'problem' and some speculation, you might need to check my 'workings'.

You are interested in evidence. Fisher proposed the use of p values as evidence but the evidence within a dataset against the null hypothesis is more readily (sensibly?) shown with a likelihood function than the p value. However, a more extreme p value is stronger evidence.

This is my solution: Don't use the Welch's t-test, but instead transform the data with a square-root transform to equalise the variances and then use a standard Student's t-test. That transform works well on your data and is one of the standard approaches for data that is heteroscedastic. The order of the p values now matches your intuition and will serve for evidence.

If you are using p values as evidence rather than attempting to protect against long-term false positive errors then the arguments for adjusting the p values for multiple comparisons become fairly weak, in my opinion.

Now, the speculative part. As I understand it, Welch's t-test is a solution to the Fisher-Behrens problem (testing means where the data have unequal variances), but it is a solution that Fisher was unhappy with. Perhaps it is a Neyman-Pearsonian in its underlying philosophy. Anyway, the amount of evidence in a p value from a t-test is dependent on the p value AND on the sample size. (That is not widely recognised, perhaps because the evidence in a p value from a z-test is independent of the sample size.) I suspect that the Welch's test screws up the evidential nature of the p value by its adjustment of degrees of freedom.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for pointing out the square root transformation as a solution. I'll look into it. $\endgroup$
    – ALiX
    May 9, 2012 at 16:50
  • $\begingroup$ (cont'd) I don't really understand your comment about p-values and multiple testing. Is there a contradiction between using p-values as evidence and adjusting for multiple testing? And your final comment about the evidence in a p-value from a t-test depending on sample size: isn't the degrees of freedom adjusting for sample sizes? And besides, how could this impact this particular data set when the sample sizes are the same for all comparisons? $\endgroup$
    – ALiX
    May 9, 2012 at 17:00
  • $\begingroup$ @AliX The evidence against the null hypothesis is quantitated best by the likelihood function. For a t-test the height of the likelihood function that corresponds to a particular p value is dependent on the sample size. In contrast, with a z-test the height of the likelihood function is unaffected by sample size. IF you are interested in evidence then I suggest you look at Statistical Evidence: A Likelihood Paradigm by Richard Royall. $\endgroup$
    – Michael Lew
    May 9, 2012 at 23:34
1
$\begingroup$

After digging around, I think my final verdict goes something like this:

To simplify the discussion, lets consider only the case when the sample sizes are equal. In that case, the approximation to the degrees of freedom can be written as

$$ \frac{\left(\frac{s_1^2}{n} + \frac{s_2^2}{n}\right)^2}{\frac{s_1^4}{n^2(n-1)} + \frac{s_2^4}{n^2(n-1)}} = ... = (n-1)\left(1 + \frac{2 s_1^2 s_2^2}{s_1^4 + s_2^4}\right), $$

where $s_1^2$ and $s_2^2$ are the sample variances and $n$ is the sample size. Hence, the degrees of freedom is $(n-1)\cdot2$ when the sample variances are equal and approaches $(n-1)$ as the sample sizes become more unequal. This means that the degrees of freedom will differ by a factor of almost 2 based only on the sample variances. Even for reasonably-sized sample sizes (say 10 or 20) the situation illustrated in the main post can easily occur.

When many t-tests are performed, sorting the comparisons by p-value could easily result in the best comparisons not making it to the top of the list, or being excluded after adjusting for multiple testing.

My personal opinion is that this is a fundamental flaw in Welch's t-test since it is designed for comparisons between samples with unequal variances, yet the more unequal the variances become, the more you lose power (in the sense that the ordering of your p-values will be wrong).

The only solution I can think of is to either use some permutation-based testing instead or transform the data so that the variances in your tests are not too far from each other.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I dont think it's should be called "fundamental flaw." Everything's relative to something. Welch's t-test came out in response to underestimated type I error of t-test of spooled variance, so it's an improved way of controlling type I error "compared to" spooled variance t-test. But when it comes to MCP and very low p-values, it sure has problems. $\endgroup$
    – KH Kim
    May 12, 2012 at 2:14
  • 2
    $\begingroup$ How are you going to organize your permutations? If the samples really come from populations which differ in variance then even under the null, the group labels are not arbitrary - at equal $n$, if a value is close to the mean it's much more likely to have come from the group with smaller variance. So you don't seem to be able to make the argument that you can just permute labels under the null. $\endgroup$
    – Glen_b
    Oct 21, 2014 at 0:17
0
$\begingroup$

As far as I know, I heard Welch's t-test which use the Satterthwaite approximation

is verified for 0.05 significance test.

Which means when P(linear combination of chi-squared distribuiton > c)=0.05,

we can get approximate c.

So, I think p-value is quite reliable around 0.05,

And obviously it's not so when it gets much less than 0.05.

p1=0 p2=0 for (m in 1:50) { a<-c(-m+95.47, -m+87.90, -m+99.00) c<-c(38.4, 40.4, 32.8) d<-c(1.8, 1.2, 1.1) p1[m]=t.test(a,c, var.eqaul=F)$p.value p2[m]=t.test(a,d, var.eqaul=F)$p.value } plot(1:50, p1, col="black") points(1:50, p2, col="red")

You can see the p-values get more correct as it approaches 0.05...

So We must not use p-values which is much less than 0.05 when using Welch's t-test.

If it is used, I think we should write a paper about it.

Anyhow, I am currently writing about "Statistics" and this theme is intriguing.

I hope to use your data writing the book with your permission.

Would you let me use your data?

And I will be grateful if you could tell the source of data and the context from which

they came!

Improve this answer
$\endgroup$
2
  • $\begingroup$ I did some research on this and I found the approximation goes like this. You want to know the distribution of a chi_1^2(df1)+ b chi_2^2(df2) but the exact distribution is so complicated so the approximation kicks in. And let a chi_1^2 + b chi_2^2 = c chi_3^2(df3), and set df3 so that the averages and the second moments of two distribution is the same. so the p=0.5 is exact but as it goes farther away from it the difference b2 the exact p and approximate p gets larger. I remember when Welch's t was performed, R always printed "p-value is not exact" I guess $\endgroup$
    – KH Kim
    May 11, 2012 at 21:42
  • $\begingroup$ I am only analyzing the data, and as such, the data don't actually belong to me. But once the data is published (should be soon) you should be able to use it as you like. $\endgroup$
    – ALiX
    May 12, 2012 at 0:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.