5
$\begingroup$

Let’s say I have a high number of treatments (40) but a low number of replications per treatment (n=4), and I am interested in comparing values between two particular treatments (let’s say treatments A and B).

The estimated variances for A and B are unequal, so I use a t-test with unpooled variances. Because of the low number of observations for each population though, it’s not very strong test.

However, there is a linear relationship between the estimated standard deviations and the estimated means of the forty treatments. Example

And I was wondering if there is a way use this information to more powerfully discern a difference between A and B.

In the simple t-test above, I have relatively little confidence in the estimated standard deviations used because of the low number of replicates. However, I wonder if I don’t have more usable information available to me than is being incorporated in this simple t-test.

Specifically, I am wondering if the linear relationship between the estimated standard deviations and the estimated means could be used to estimate standard deviation with a higher degree of certainty than we can get with just the 4 data points for each treatment alone?

If so, can this certainty somehow be translated into greater statistical power when comparing means?

$\endgroup$

2 Answers 2

1
$\begingroup$

You should be able to make use of the information that standard deviation is related to mean, and it may yield greater power ... but I wouldn't use a t-test for that.

However, you've already tested the data; consider what would have happened if you'd instead had a nice significant result -- would you really have done any more than the unpooled t-test*?

*(presumably Welch-Satterthwaite)

If the honest answer is you'd have stopped there$^\dagger$, then to start looking for some more powerful procedure now is significance-hunting, plain and simple, and if we account for the effect of that on p-values -- well, your p-values will go back up again.

$\dagger$ (on the other hand, if you'd claim you would have searched for a better test even if it was highly significant, then what the heck were you doing calculating it in the first place? Figure out what you're doing before you start playing with the data to see how significant it is -- what else is anyone to make of it when you do?)

Leaving aside the significance-hunting for the moment, there are several ways I might think about this; here are three:

  1. use a Gamma GLM (which has sd proportional to mean built into the model);

  2. take logs (which should yield almost constant standard-deviation); this means you're comparing means-of-logs when you do a t-test, which corresponds to a scale-shift alternative hypothesis (as does the GLM above);

  3. alternatively if you're looking for fairly general (tend-to-be-bigger) alternatives you could apply a Wilcoxon-Mann-Whitney test on the original scale (there's no need to transform, it already encompasses the effect of monotonic transforms).

However, with 4 observations per group, you have low power, and no amount of cleverness can do much about it. You can improve your estimate of the error-variance by (i) up-front focusing attention on the A vs B comparison alone, but (ii) combining the data into an analysis where their information can be used to get a better handle on the variability, which should improve power. In the case of 2. above you'd use a pre-specified A-B contrast in the context of an ANOVA; in 1. above you'd set up something similar in the GLM.

In the comparison of ANOVA vs a t-test (under option 2.) this should improve error df from 6 to about 30, which at 5% significance level on your particular contrast will be like reducing standard errors by 1/7.

$\endgroup$
3
  • $\begingroup$ On the significance-hunting note--I hadn’t really thought about that before, and you’re right. If I had found the expected significance, I probably wouldn’t have given the choice in test a second thought. However, as to picking a better test to begin with, I simply didn’t/don't know what tools are out there (being armed with very little statistical training). That being said, I’m not familiar with the all the terms/tools in your answer, but I will start looking them up and trying the approaches you suggest (as an educational endeavor not a significance hunting one, I promise!). Many thanks! $\endgroup$
    – Angela
    Jun 28, 2015 at 16:06
  • $\begingroup$ If there's something specific about my answer you'd like explained, just ask. If a short explanation is possible here, I'll give it (a long explanation may require a new question). As a general principle, you should try to choose your testing strategy before you even have the data. Often you can (either from thinking about the situation, or discover by reading other studies, talking to experts, etc) infer enough about a particular variable's general features to know what you need. (E.g. variables that can only be positive are likely to be skew and have increasing spread with increasing mean) $\endgroup$
    – Glen_b
    Jun 29, 2015 at 0:25
  • 1
    $\begingroup$ I sure will--though the question(s) will be a bit delayed. My opportunities to delve into these issues come in short, infrequent bursts (mainly when I have someone to help me watch after my little one). Thanks so much! $\endgroup$
    – Angela
    Jun 30, 2015 at 21:18
-1
$\begingroup$

No.

All skewed distributions exhibit this behavior to some extent. For example, I followed the exact specifications of your test. Each treatment has four replications and there are 40 treatments. The thing is, they are all pulled from the same distribution (I based it off a normal distribution, but tried to make it match your data).

The mean of the population from which all the sample were drawn is about 70, which means that the subpopulation means should not be statistically significantly different from each other (at least most of them).

Here is the code to generate the simulation:

%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

np.random.seed(111)

replications = 4
number_of_treatments = 40
means = []
standard_deviations = []

for _ in xrange(number_of_treatments):
    treatment = abs(np.random.normal(loc=1.5, size=replications))*20+40
    means.append(treatment.mean())
    standard_deviations.append(treatment.std())

m, b = np.polyfit(means, standard_deviations, 1)
plt.scatter(means, standard_deviations)
plt.plot(means, m*np.array(means)+b, '-')
plt.xlabel("Means of each treatment")
plt.ylabel("Standard deviation of each treatment")
plt.show()

Here is the plot:

Similar example

Notice that it is nearly identical to your distribution (the slope isn't as extreme as yours, but that was just because it was difficult getting everything to match).

Here is another simulation where the distribution is even more skewed and is based on abs(np.random.normal(loc=0, size=replications))**2.

More extreme slope

Bottom line is that your results can be exactly replicated even though all of the treatments are pulled from the same distribution. Any "technique" that uses the trend in standard deviation to separate one result from the other (no matter what technique it is) would also be just as easily applied to the data from this simulation to generate obviously erroneous results.

So the relationship is simply a reflection of the distribution, not a difference in treatments.

Also, be careful with the multiple comparisons problem. If you do enough tests, you are expected to find some that are "statistically significant" just by chance.

$\endgroup$
2
  • 1
    $\begingroup$ If you fit a glm that includes a mean variance relation, you should have much more power than fitting a bunch on unpooled t-tests $\endgroup$
    – Cliff AB
    Jun 27, 2015 at 0:30
  • $\begingroup$ My weak understanding of statistics may be hindering me here, but if I understand correctly, you're saying that it’s possible for a single population to yield a linear relationship between standard deviation and mean when sampled repeatedly--which would mean that using this relationship to help separate groups could yield incorrect results? If I’ve understood that correctly, then I wonder how probable this situation is because in any test, I thought we accepted the chance that improbable but possible data might lead us astray. I’ll have to think on this some more. Thanks for pointing this out! $\endgroup$
    – Angela
    Jun 28, 2015 at 16:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.