What to do when the means of two samples are significantly different but the difference seems too small to matter

Question

I have two samples ($n \approx 70$ in both cases). The means differ by about twice the pooled std. dev. The resulting $T$ value is approximately 10. Whilst it's great to know that I have conclusively shown that the means are not the same, this seems to me to be driven by the large n. Looking at histograms of the data I certainly do not feel that such as small p-value is really representative of the data and, to be honest, don't really feel comfortable quoting it. I'm probably asking the wrong question. What I'm thinking is: ok, the means are different but does that really matter as the distributions share a significant overlap?

Is this where Bayesian testing is useful? If so, where is a good place to start, a bit of googling hasn't yielded anything useful but I may not by asking the right question. If this is the wrong thing does anyone have any suggestions? Or is this simply a point for discussion as opposed to quantitative analysis?

Answer 1

Let $\mu_1$ denote the mean of the first population and $\mu_2$ denote the mean of the second population. It seems that you've used a two-sample $t$-test to test whether $\mu_1=\mu_2$. The significant result implies that $\mu_1\neq\mu_2$, but the difference seems to be to small to matter for your application.

What've you encountered is the fact that statistically significant often can be something else than significant for the application. While the difference may be statistically significant it may still not be meaningful.

Bayesian testing won't solve that problem - you'll still just conclude that a difference exists.

There might however be a way out. For instance, for a one-sided hypothesis you could decide that if $\mu_1$ is $\Delta$ units greater than $\mu_2$ then that would be a meaningful difference that is large enough to matter for your application.

In that case you would test whether $\mu_1-\mu_2\leq \Delta$ instead of whether $\mu_1-\mu_2=0$. The $t$-statistic (assuming equal variances) would in that case be $$ T=\frac{\bar{x}_1-\bar{x}_2-\Delta}{s_p\sqrt{1/n_1+1/n_2}}$$ where $s_p$ is the pooled standard deviation estimate. Under the null hypothesis, this statistic is $t$-distributed with $n_1+n_2-2$ degrees of freedom.

An easy way of carrying out this test is to subtract $\Delta$ from your observations from the first population and then carry out a regular one-sided two-sample $t$-test.

Answer 2

It is valid to compare several approaches, but not with the aim of choosing the one that favours our desires/believes.

My answer to your question is: It is possible that two distributions overlap while they have different means, which seems to be your case (but we would need to see your data and context in order to provide a more precise answer).

I am going illustrate this using a couple of approaches for comparing normal means.

1. $t$-test

Consider two simulated samples of size $70$ from a $N(10,1)$ and $N(12,1)$, then the $t$-value is approximately $10$ as in your case (See the R code below).

rm(list=ls())
# Simulated data
dat1 = rnorm(70,10,1)
dat2 = rnorm(70,12,1)

set.seed(77)

# Smoothed densities
plot(density(dat1),ylim=c(0,0.5),xlim=c(6,16))
points(density(dat2),type="l",col="red")

# Normality tests
shapiro.test(dat1)
shapiro.test(dat2)

# t test
t.test(dat1,dat2)

However the densities show a considerable overlapping. But remember that you are testing a hypothesis about the means, which in this case are clearly different but, due to the value of $\sigma$, there is an overlap of the densities.

2. Profile likelihood of $\mu$

For a definition of the Profile likelihood and likelihood please see 1 and 2.

In this case, the profile likelihood of $\mu$ of a sample of size $n$ and sample mean $\bar{x}$ is simply $R_p(\mu)=\exp\left[-n(\bar{x}-\mu)^2\right]$.

For the simulated data, these can be calculated in R as follows

# Profile likelihood of mu
Rp1 = function(mu){
n = length(dat1)
md = mean(dat1)
return( exp(-n*(md-mu)^2) )
}

Rp2 = function(mu){
n = length(dat2)
md = mean(dat2)
return( exp(-n*(md-mu)^2) )
}

vec=seq(9.5,12.5,0.001)
rvec1 = lapply(vec,Rp1)
rvec2 = lapply(vec,Rp2)

# Plot of the profile likelihood of mu1 and mu2
plot(vec,rvec1,type="l")
points(vec,rvec2,type="l",col="red")

As you can see, the likelihood intervals of $\mu_1$ and $\mu_2$ do not overlap at any reasonable level.

3. Posterior of $\mu$ using Jeffreys prior

Consider the Jeffreys prior of $(\mu,\sigma)$

$$\pi(\mu,\sigma)\propto \dfrac{1}{\sigma^2}$$

The posterior of $\mu$ for each data set can be calculated as follows

# Posterior of mu
library(mcmc)

lp1 = function(par){
n=length(dat1)
if(par[2]>0) return(sum(log(dnorm((dat1-par[1])/par[2])))- (n+2)*log(par[2]))
else return(-Inf)
}

lp2 = function(par){
n=length(dat2)
if(par[2]>0) return(sum(log(dnorm((dat2-par[1])/par[2])))- (n+2)*log(par[2]))
else return(-Inf)
}

NMH = 35000
mup1 = metrop(lp1, scale = 0.25, initial = c(10,1), nbatch = NMH)$batch[,1][seq(5000,NMH,25)]
mup2 = metrop(lp2, scale = 0.25, initial = c(12,1), nbatch = NMH)$batch[,1][seq(5000,NMH,25)]

# Smoothed posterior densities
plot(density(mup1),ylim=c(0,4),xlim=c(9,13))
points(density(mup2),type="l",col="red")

Again, the credibility intervals for the means do not overlap at any reasonable level.

In conclusion, you can see how all these approaches indicate a significant difference of means (which is the main interest), despite the overlapping of the distributions.

$\star$ A different comparison approach

Judging by your concerns about the overlapping of the densities, another quantity of interest might be ${\mathbb P}(X<Y)$, the probability that the first random variable is smaller than the second variable. This quantity can be estimated nonparametrically as in this answer. Note that there are no distributional assumptions here. For the simulated data, this estimator is $0.8823825$, showing some overlap in this sense, while the means are significantly different. Please, have a look to the R code shown below.

# Optimal bandwidth
h = function(x){
n = length(x)
return((4*sqrt(var(x))^5/(3*n))^(1/5))
}

# Kernel estimators of the density and the distribution
kg = function(x,data){
hb = h(data)
k = r = length(x)
for(i in 1:k) r[i] = mean(dnorm((x[i]-data)/hb))/hb
return(r )
} 

KG = function(x,data){
hb = h(data)
k = r = length(x)
for(i in 1:k) r[i] = mean(pnorm((x[i]-data)/hb))
return(r ) 
} 

# Baklizi and Eidous (2006) estimator
nonpest = function(dat1B,dat2B){
return( as.numeric(integrate(function(x) KG(x,dat1B)*kg(x,dat2B),-Inf,Inf)$value))  
}

nonpest(dat1,dat2)

I hope this helps.

Answer 3

Answering the Right Question

ok, the means are different but does that really matter as the distributions share a significant overlap?

Any test that asks whether group means are different will, when it works right, tell you whether means are different. It will not tell you that distributions of the data itself are different, since that is a different question. That question certainly depends on the whether the means are different but also on many other things that might be (incompletely) summarised as variance, skew, and kurtosis.

You correctly note that certainty about where the means are depends on the amount of data you have to estimate them, so having more data will allow you spot mean differences in more nearly overlapping distributions. But you wonder whether

such as small p-value is really representative of the data

Indeed it is not, at least not directly. And this is by design. It is representative (approximately speaking) of the certainty you can have that a particular pair of sample statistics of the data (not the data itself) are different.

If you wanted to represent the data itself in a more formal way than simply showing the histograms and testing moments of it, then perhaps a pair of density plots might be helpful. It rather depends really on the argument you are using the test to make.

A Bayesian Version

In all these respects, Bayesian difference 'tests' and T-tests will behave the same way because they are trying to do the same thing. The only advantages I can think of for using a Bayesian approach are: a) that it will be easy to do the test allowing possibly different variances for each group, and b) that it will focus on estimating the probable size of the difference in means rather than finding a p-value for some test of difference. That said, these advantages are pretty minor: e.g. in b) you could always report a confidence interval for the difference.

The quote marks above over 'tests' are deliberate. It is certainly possible to do Bayesian hypothesis testing, and people do. However, I would suggest that the comparative advantage of the approach is in the focus on building a plausible model of the data and communicating its important aspects with appropriate levels of uncertainty.

Answer 4

First of all, this is not a problem to pin on frequentist testing. The problem lies in the null hypothesis that the means are exactly equal. Therefore if the populations differ in means by any small amount and the sample size is large enough the chance to reject this null hypothesis is very high. Therefore the p-value for your test turned out to be very small. The culprit is the choice of the null hypothesis. Pick d>0 and take the null hypothesis to be that the means differ by less than d in absolute value. You pick d so that the real difference has to be satisfactorily large to reject. Your problem goes away. Bayesian testing does not solve your problem if you insist on a null hypothesis of exact equality of means.