1
$\begingroup$

I want to test if the means of two groups of measurements are equal or not. I perform the analysis using R software and, in particular, the function t.test(). Even though the means of the two groups are different (the mean of group A is -0.04570781 and the mean of group B is 0.03339135) , the p-value is higher than the significance threshold 0.05 (it's 0.213). Why does it happen ? The group A has 95 measurements, while the group B only 10. Is it due to the different sample size of the two groups of measurements ? Or maybe the difference between the values of the two means is too low to be detected ?

The output results of my test is

Welch Two Sample t-test

data:  x and y
t = -1.3223, df = 10.98, p-value = 0.213
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.21079297  0.05259464
sample estimates:
  mean of x   mean of y 
-0.04570781  0.03339135

Moreover my code is:

x <- RadVsNotRadiation[RadVsNotRadiation$Condition=='Irradiated',"mRNA"]
y <- RadVsNotRadiation[RadVsNotRadiation$Condition=='reference',"mRNA"]
t.test(x,y)

and the data that I have used are:

> dput(RadVsNotRadiation)
structure(list(Gene = c("ID-1", "ID-1", "ID-1", "ID-1", "ID-1", 
"ID-1", "ID-1", "ID-1", "ID-1", "ID-1", "ID-1", "ID-1", "ID-1", 
"ID-1", "ID-1", "ID-1", "ID-1", "ID-1", "ID-1", "ID-4", "ID-4", 
"ID-4", "ID-4", "ID-4", "ID-4", "ID-4", "ID-4", "ID-4", "ID-4", 
"ID-4", "ID-4", "ID-4", "ID-4", "ID-4", "ID-4", "ID-4", "ID-4", 
"ID-4", "ID-5", "ID-5", "ID-5", "ID-5", "ID-5", "ID-5", "ID-5", 
"ID-5", "ID-5", "ID-5", "ID-5", "ID-5", "ID-5", "ID-5", "ID-5", 
"ID-5", "ID-5", "ID-5", "ID-5", "ID-6", "ID-6", "ID-6", "ID-6", 
"ID-6", "ID-6", "ID-6", "ID-6", "ID-6", "ID-6", "ID-6", "ID-6", 
"ID-6", "ID-6", "ID-6", "ID-6", "ID-6", "ID-6", "ID-6", "ID-7", 
"ID-7", "ID-7", "ID-7", "ID-7", "ID-7", "ID-7", "ID-7", "ID-7", 
"ID-7", "ID-7", "ID-7", "ID-7", "ID-7", "ID-7", "ID-7", "ID-7", 
"ID-7", "ID-7", "ID-1", "ID-1", "ID-4", "ID-4", "ID-5", "ID-5", 
"ID-6", "ID-6", "ID-7", "ID-7"), mRNA = c(-0.181385669, -0.059647494, 
0.104476117, NA, NA, NA, -0.052190978, -0.040484945, 0.194226742, 
-0.501601326, 0.102342605, -0.127143845, -0.008523742, -0.102946211, 
-0.042894028, 0.002922923, -0.134394347, -0.214204393, NA, -0.138122686, 
0.203242361, 0.097935502, NA, NA, NA, 0.147068146, -0.089430917, 
0.331565412, -0.034572422, -0.129896329, 0.324191, 0.470108479, 
-0.027268223, 0.232304713, 0.090348708, 0.070848402, 0.181540708, 
-0.502255367, -0.267631441, -0.368647839, -0.040910404, -0.003983171, 
-0.003983171, -0.003983171, -0.14980589, -0.119449612, -0.309154214, 
-0.487589361, 0.272803506, -0.421733575, NA, -0.467108567, 0.024868338, 
-0.156025729, -0.044680175, -0.206716896, -0.272014193, -0.230499883, 
-0.238597397, -0.118130949, 0.349957464, 0.349957464, 0.349957464, 
0.172048587, -0.186226994, 0.16113822, -0.293029136, -0.111636253, 
-0.044189887, 0.081555274, -0.048106079, -0.05853566, 0.010407814, 
-0.066981809, -0.09828484, NA, -0.315190986, -0.005102456, 0.221556197, 
0.206584568, 0.206584568, 0.206584568, 0.102649006, NA, -0.011777384, 
-0.36963487, -0.054853074, -0.230240699, -0.210508323, -0.208889919, 
-0.050763372, 0.023073782, -0.095118984, -0.091076071, -0.330257395, 
0.102772933, 0.247872038, 0.216357646, 0.126169901, -0.237278842, 
-0.066908278, 0.105082639, NA, -0.050061512, -0.143484352), Condition = structure(c(1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 
1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 1L, 2L, 2L, 
2L, 2L, 2L, 2L, 2L, 2L, 2L, 2L), .Label = c("Irradiated", "reference"
), class = "factor")), row.names = c(NA, -105L), class = "data.frame")

Any suggestion will be appreciated. Thank you in advance.

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Can you edit the question and paste in the t.test results, it would be easier to provide an answer seeing the statement and the output. Yes, the variability of one or both groups could be large enough to that a 0.08 difference could be a result of chance alone. $\endgroup$
    – Dave2e
    Commented Mar 20, 2021 at 13:45

3 Answers 3

Reset to default
4
$\begingroup$

This is a good time to talk about false negatives and statistical power.

A false positive is when you falsely reject the null (i.e. claim a difference exists). A false negative is when you fail to reject the null when you should (i.e. when a difference exists, but you say there is no difference).

What reasons would cause us to commit a false negative (sometimes called a Type II error)? Well, the data are modelled as random and so it might just be that we got lucky and captured data which did not show a difference. Here is R code with an example of that

set.seed(1)

x = rnorm(10, 1, 2)
y = rnorm(10, 0, 2)
t.test(x,y)
> t.test(x,y, var.equal = T)

    Two Sample t-test

data:  x and y
t = 0.91557, df = 18, p-value = 0.372
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.9926364  2.5260676
sample estimates:
mean of x mean of y 
1.2644056 0.4976899

The means of the two groups really are different (the mean of x is 1, the mean of y is 0) but we fail to reject the null. But, if I change the random seed to 0 then I reject the null!

So clearly, there is a probabilistic element to rejecting the null. The probability I reject the null when it is false is called statistical power. You can read up on the power of the t test on your own, but there are three main things which effect power:

  • The sample size. If you have lots of data, you (usually) have high power. This is because we can estimate the means with high precision, meaning we are more certain about the means in each group. For the t test, the most power comes from when the sample sizes are equal.

  • The noise in the outcome. Noisier outcomes (usually) lower our statistical power because the noise makes it harder to estimate the mean.

  • The size of the difference between groups. Even if we have small samples and lots of noise, of the difference is really big then we will have high power. Its easier to detect differences when they are very clear.

In light of this, let's examine why you might have failed to reject the null.

  • Your sample sizes are not equal. Remember, optimal power comes from equal sample sizes.

  • The difference is too small. The size of the difference is approx 0.08. That is a very very small difference. Now, I don't know how much noise is in your data but if there is even a small amount of noise (like a standard deviation of 0.25) then your signal may be washed out by the noise.

  • You may just be unlucky.

Can you post the results of your t test? That can help is do a post mortem.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you so much @Demetri Papanos . Your answer is already very helpful but if you want to have a look at my results, a post morted on the basis of my specific situation would be great ! I have modified the post $\endgroup$
    – HelpNeederStudent
    Commented Mar 20, 2021 at 14:31
  • 1
    $\begingroup$ @Manuela It is difficult to say for sure. That you only have 10 observations in the one group makes it all that harder to diagnose, but I suspect the top reason is the imbalance of sample size leading to small precision in the estimates of the group means. $\endgroup$
    – Demetri Pananos
    Commented Mar 20, 2021 at 14:50
  • $\begingroup$ Ok thank you again... that's reasonable. I have a last question: do you know if there is some function in R that can give you like an estimate of the error due to the imbalance of the sample size of the two groups ? (I was looking at the t.test itsself but I do not find anything) $\endgroup$
    – HelpNeederStudent
    Commented Mar 20, 2021 at 15:06
  • $\begingroup$ @Manuela You might want to look at the standard error of the difference. This quantity can be found in most intro stats books. It would be the denominator of a t test. $\endgroup$
    – Demetri Pananos
    Commented Mar 20, 2021 at 15:17
  • $\begingroup$ Ok @Demetri Pananos I will think about it . Thank you . $\endgroup$
    – HelpNeederStudent
    Commented Mar 20, 2021 at 15:23
2
$\begingroup$

Other answers make the main points well, but in this case looking at the data does no harm and should be complementary.

The major issues, as said, are the smallness of one sample and the small size of the difference between means combined with variability in both samples. The significance test is unsurprising in this light.

The graph is a quantile-box plot with quantile plot (data points in rank order) and fairly conventional median and quartiles boxes. The extra horizontal lines show the means. As all the data are shown, there is no need for extra whiskers or whatnot.

enter image description here

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you @Nick Cox . This also helps ! $\endgroup$
    – HelpNeederStudent
    Commented Mar 20, 2021 at 15:01
0
$\begingroup$

The short and simple answer is that the whole purpose of a t test (and significance testing in general) is to tell you how "confident" you can be that two means that LOOK different in your sample actually ARE different in the population. The fact that the t test is not significant in your case means that, even though the means of A and B look different in your dataset, you don't have enough data to be confident that this difference reflects an actual difference between A and B in the population, rather than just random error, due to the fact that (we assume) your data on A and B are from a random sample, and not the entire population.

Now, WHY is the t test not significant here? Well it could be because there really is no difference between A and B in the larger population and the fact that the two means look different in your sample is just random sampling error. Or it could be that A and B really are different in the population but because you only have a few observations (only 10 for B) you just don't have enough data to be confident that the difference you see is not random error. But without more data there is no way to distinguish between these two scenarios, so in this case we are usually forced to say something like "the difference in means between A and B was not statistically significant" to reflect the fact that we just don't know whether A and B are different or not, given the data we have.

Improve this answer
$\endgroup$
4
  • $\begingroup$ ok @Graham Wright, thank you. But I do not understand a point: the unbalance in the sample size of the two groups should be taken into account in the math of the t.test function when it does a Welch two sample t-test, right ? So in principle I thought that there should be no problems with that... $\endgroup$
    – HelpNeederStudent
    Commented Mar 20, 2021 at 14:39
  • $\begingroup$ This is the gist of it. But I fear that talk of confidence can drift into seeming to imply what a researcher new to this is supposed to feel. As you know, the point of a significance test is to produce an objective calculation of a P-value, guiding a conclusion if not a decision, and is nothing to do with the researcher's state of mind. The problem is several decades old, that Neyman took an existing English word confidence and gave it a technical meaning, whereas it is just a term of art. (I would not prefer some different jargon such as C-interval, but the problem remains.) $\endgroup$
    – Nick Cox
    Commented Mar 20, 2021 at 16:43
  • $\begingroup$ @Manuela - the t test takes into account the sample size (and standard deviations) of both estimates, and in your case it has determined that the sample size (at least for B, but maybe for both) is too small to detect a significant difference. The unbalanced sample size isn't a "problem" for the t test, it is just giving you the "correct" answer given the data you provided it: despite the fact that these means are different in your dataset, you do not have enough data to reject the null hypotheses that these two means are actually the same in the underlying population. $\endgroup$
    – Graham Wright
    Commented Mar 21, 2021 at 13:47
  • $\begingroup$ Thank you that's clear ! $\endgroup$
    – HelpNeederStudent
    Commented Mar 22, 2021 at 11:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.