Can a linear relationship between estimated standard deviations and estimated means be used to increase the power of a t-test?

Question

Let’s say I have a high number of treatments (40) but a low number of replications per treatment (n=4), and I am interested in comparing values between two particular treatments (let’s say treatments A and B).

The estimated variances for A and B are unequal, so I use a t-test with unpooled variances. Because of the low number of observations for each population though, it’s not very strong test.

However, there is a linear relationship between the estimated standard deviations and the estimated means of the forty treatments.

And I was wondering if there is a way use this information to more powerfully discern a difference between A and B.

In the simple t-test above, I have relatively little confidence in the estimated standard deviations used because of the low number of replicates. However, I wonder if I don’t have more usable information available to me than is being incorporated in this simple t-test.

Specifically, I am wondering if the linear relationship between the estimated standard deviations and the estimated means could be used to estimate standard deviation with a higher degree of certainty than we can get with just the 4 data points for each treatment alone?

If so, can this certainty somehow be translated into greater statistical power when comparing means?

Answer 1

You should be able to make use of the information that standard deviation is related to mean, and it may yield greater power ... but I wouldn't use a t-test for that.

However, you've already tested the data; consider what would have happened if you'd instead had a nice significant result -- would you really have done any more than the unpooled t-test*?

*(presumably Welch-Satterthwaite)

If the honest answer is you'd have stopped there$^\dagger$, then to start looking for some more powerful procedure now is significance-hunting, plain and simple, and if we account for the effect of that on p-values -- well, your p-values will go back up again.

$\dagger$ (on the other hand, if you'd claim you would have searched for a better test even if it was highly significant, then what the heck were you doing calculating it in the first place? Figure out what you're doing before you start playing with the data to see how significant it is -- what else is anyone to make of it when you do?)

Leaving aside the significance-hunting for the moment, there are several ways I might think about this; here are three:

1. use a Gamma GLM (which has sd proportional to mean built into the model);

2. take logs (which should yield almost constant standard-deviation); this means you're comparing means-of-logs when you do a t-test, which corresponds to a scale-shift alternative hypothesis (as does the GLM above);

3. alternatively if you're looking for fairly general (tend-to-be-bigger) alternatives you could apply a Wilcoxon-Mann-Whitney test on the original scale (there's no need to transform, it already encompasses the effect of monotonic transforms).

However, with 4 observations per group, you have low power, and no amount of cleverness can do much about it. You can improve your estimate of the error-variance by (i) up-front focusing attention on the A vs B comparison alone, but (ii) combining the data into an analysis where their information can be used to get a better handle on the variability, which should improve power. In the case of 2. above you'd use a pre-specified A-B contrast in the context of an ANOVA; in 1. above you'd set up something similar in the GLM.

In the comparison of ANOVA vs a t-test (under option 2.) this should improve error df from 6 to about 30, which at 5% significance level on your particular contrast will be like reducing standard errors by 1/7.

Answer 2

No.

All skewed distributions exhibit this behavior to some extent. For example, I followed the exact specifications of your test. Each treatment has four replications and there are 40 treatments. The thing is, they are all pulled from the same distribution (I based it off a normal distribution, but tried to make it match your data).

The mean of the population from which all the sample were drawn is about 70, which means that the subpopulation means should not be statistically significantly different from each other (at least most of them).

Here is the code to generate the simulation:

%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

np.random.seed(111)

replications = 4
number_of_treatments = 40
means = []
standard_deviations = []

for _ in xrange(number_of_treatments):
    treatment = abs(np.random.normal(loc=1.5, size=replications))*20+40
    means.append(treatment.mean())
    standard_deviations.append(treatment.std())

m, b = np.polyfit(means, standard_deviations, 1)
plt.scatter(means, standard_deviations)
plt.plot(means, m*np.array(means)+b, '-')
plt.xlabel("Means of each treatment")
plt.ylabel("Standard deviation of each treatment")
plt.show()

Here is the plot:

Notice that it is nearly identical to your distribution (the slope isn't as extreme as yours, but that was just because it was difficult getting everything to match).

Here is another simulation where the distribution is even more skewed and is based on abs(np.random.normal(loc=0, size=replications))**2.

Bottom line is that your results can be exactly replicated even though all of the treatments are pulled from the same distribution. Any "technique" that uses the trend in standard deviation to separate one result from the other (no matter what technique it is) would also be just as easily applied to the data from this simulation to generate obviously erroneous results.

So the relationship is simply a reflection of the distribution, not a difference in treatments.

Also, be careful with the multiple comparisons problem. If you do enough tests, you are expected to find some that are "statistically significant" just by chance.