Welch's t-test gives worse p-value for more extreme difference Here are four different sets of numbers:
A = {95.47, 87.90, 99.00}
B = {79.2, 75.3, 66.3}
C = {38.4, 40.4, 32.8}
D = {1.8, 1.2, 1.1}  
Using a two-sample t-test without assuming equal variances, I compare B, C, and D to A and get the following p-values:
0.015827  (A vs B)
0.000283  (A vs C)
0.001190  (A vs D)  
I find it strange that the p-value from the A-D test is worse than the A-C test: the difference between the means is clearly much bigger AND the variance of D is much lower than the variance of C. Intuitively (at least for my intuition), both these facts should drive the p-value lower.
Could someone explain if this is a desired or expected behavior of the t-test or whether it has to do more with my particular data set (extreme low sample size perhaps?). Is the t-test inappropriate for this particular set of data?
From a purely computational point of view, the reason for a worse p-value seems to be the degrees of freedom, which in the A-D comparison is 2.018 while it is 3.566 in the A-C comparison. But surely, if you just saw those numbers, wouldn't you think that there is stronger evidence for rejecting the null hypothesis in the A-D case compared to A-C?
Some might suggest that this is not a problem here since all p-values are quite low anyway. My problem is that these 3 tests are part of a suite of tests that I am performing. After correcting for multiple testing, the A-D comparison doesn't make the cut, while the A-C comparison does. Imagine plotting those numbers (say bar-plots with error bars as biologists often do) and trying to justify why C is significantly different from A but D is not... well, I can't.
Update: why this is really important
Let me clarify why this observation could have a great impact on interpreting past studies. In bioinfomatics, I have seen the t-test be applied to small sample sizes on a large scale (think differential gene expression of hundreds or thousands of genes, or the effect of many different drugs on a cell line, using only 3-5 replicates). The usual procedure is to do many t-tests (one for each gene or drug) followed by multiple testing correction, usually FDR. Given the above observation of the behaviour of Welch's t-test, this means that some of the very best cases are being systematically filtered out. Although most people will look at the actual data for the comparisons at the top of their list (the ones with best p-values), I don't know of anyone who will look through the list of all comparisons where the null hypothesis wasn't rejected.
 A: Yes, it's the degrees of freedom. The t-statistics themselves increase as we compare groups B,C,D to A; the numerators get bigger and the denominators get smaller. 
Why doesn't your approach "work"? Well, the Satterthwaite approximation for the degrees of freedom, and the reference distribution is (as the name suggests!) just an approximation. It would work fine if you had more samples in each group, and not hugely heavy-tailed data; 3 observations per group is really very small for most purposes. (Also, while p-values are useful for doing tests, they don't measure evidence and don't estimate parameters with direct interpretations in terms of data.)
If you really want to work out the exact distribution of the test statistic - and a better calibrated p-value - there are methods cited here that could be used. However, they rely on assuming Normality, an assumption you have no appreciable ability to check, here.
A: There is quite a bit to this question, and I'm fairly sure that some of it is beyond my understanding. Thus while I have a likely solution to the 'problem' and some speculation, you might need to check my 'workings'.
You are interested in evidence. Fisher proposed the use of p values as evidence but the evidence within a dataset against the null hypothesis is more readily (sensibly?) shown with a likelihood function than the p value. However, a more extreme p value is stronger evidence.
This is my solution: Don't use the Welch's t-test, but instead transform the data with a square-root transform to equalise the variances and then use a standard Student's t-test. That transform works well on your data and is one of the standard approaches for data that is heteroscedastic. The order of the p values now matches your intuition and will serve for evidence.
If you are using p values as evidence rather than attempting to protect against long-term false positive errors then the arguments for adjusting the p values for multiple comparisons become fairly weak, in my opinion.
Now, the speculative part. As I understand it, Welch's t-test is a solution to the Fisher-Behrens problem (testing means where the data have unequal variances), but it is a solution that Fisher was unhappy with. Perhaps it is a Neyman-Pearsonian in its underlying philosophy. Anyway, the amount of evidence in a p value from a t-test is dependent on the p value AND on the sample size. (That is not widely recognised, perhaps because the evidence in a p value from a z-test is independent of the sample size.) I suspect that the Welch's test screws up the evidential nature of the p value by its adjustment of degrees of freedom.
A: After digging around, I think my final verdict goes something like this:
To simplify the discussion, lets consider only the case when the sample sizes are equal. In that case, the approximation to the degrees of freedom can be written as
$$
\frac{\left(\frac{s_1^2}{n} + \frac{s_2^2}{n}\right)^2}{\frac{s_1^4}{n^2(n-1)} + \frac{s_2^4}{n^2(n-1)}} = ... = (n-1)\left(1 + \frac{2 s_1^2 s_2^2}{s_1^4 + s_2^4}\right),
$$
where $s_1^2$ and $s_2^2$ are the sample variances and $n$ is the sample size. Hence, the degrees of freedom is $(n-1)\cdot2$ when the sample variances are equal and approaches $(n-1)$ as the sample sizes become more unequal. This means that the degrees of freedom will differ by a factor of almost 2 based only on the sample variances. Even for reasonably-sized sample sizes (say 10 or 20) the situation illustrated in the main post can easily occur.
When many t-tests are performed, sorting the comparisons by p-value could easily result in the best comparisons not making it to the top of the list, or being excluded after adjusting for multiple testing.
My personal opinion is that this is a fundamental flaw in Welch's t-test since it is designed for comparisons between samples with unequal variances, yet the more unequal the variances become, the more you lose power (in the sense that the ordering of your p-values will be wrong).
The only solution I can think of is to either use some permutation-based testing instead or transform the data so that the variances in your tests are not too far from each other.
A: As far as I know, I heard Welch's t-test which use the Satterthwaite approximation
is verified for 0.05 significance test.
Which means when P(linear combination of chi-squared distribuiton > c)=0.05,
we can get approximate c.
So, I think p-value is quite reliable around 0.05, 
And obviously it's not so when it gets much less than 0.05.

p1=0
p2=0
for (m in 1:50) {
  a<-c(-m+95.47, -m+87.90, -m+99.00)
  c<-c(38.4, 40.4, 32.8)
  d<-c(1.8, 1.2, 1.1)
  p1[m]=t.test(a,c, var.eqaul=F)$p.value
  p2[m]=t.test(a,d, var.eqaul=F)$p.value
}
plot(1:50, p1, col="black")
points(1:50, p2, col="red")
You can see the p-values get more correct as it approaches 0.05...
So We must not use p-values which is much less than 0.05 when using Welch's t-test.
If it is used, I think we should write a paper about it.
Anyhow, I am currently writing about "Statistics" and this theme is intriguing.
I hope to use your data writing the book with your permission.
Would you let me use your data?
And I will be grateful if you could tell the source of data and the context from which
they came!
