4
$\begingroup$

I have five independent experiments. In each experiment I record the intensity of sample A and sample B, then add these intensities together and calculate A and B as a proportion of the total. Obviously, these proportions add up to exactly 1. For example: A1=10000, B1=50000 so the proportions would be A1=0.166, B1=0.833. I do this for all my five experiments. I now have five proportions for A and five for B. The standard deviations calculated for the five A and B values, respectively, is identical. My question is a) is this an appropriate way of reporting the standard deviation? and b) is a T-test an appropriate test to see if the differences between the A and B proportions is significant?

Any help is much appreciated!

$\endgroup$

3 Answers 3

3
$\begingroup$

1) The two proportions add to 1, so they're perfectly (negatively) dependent. So you can't treat them as independent.

2) continuous proportions such as these don't have constant variance - the variance must get smaller as the proportion approaches 0 or 1.

As such, the usual independent samples, equal variance t-test assumptions are violated.

You might (perhaps) model the A proportions (say) as having something like a beta distribution, and compare them with 50%, but I think it makes more sense to come up with a suitable model for comparing A and B directly.

Presumably you're converting to proportions because the $(A_i,B_i)$ pairs tend to be quite different in size for different $i$.

  • Might a location comparison make sense on the log-scale? -- in which case perhaps a paired location test on the logs might be reasonable.

  • alternatively, might the ratios (perhaps using some GLM to model them) be of interest?

$\endgroup$
1
$\begingroup$

First it would be helpful if you use different notation for the original quantities and their normalized versions (i.e., what you refer to as proportions). I am going to use $a,b$ to stand for the normalized version of $A,B$ respectively.

Secondly, in terms of statistical test, your null hypothesis seems to be that $a=b$, in other words, $a=0.5$. One of the crucial assumptions of t-test is that the sample should follow a normal distribution. Given that $a$ is bounded between $0$ and $1$, this is patently false.

Instead, let's make the follow assumptions: $A_i$, $B_i$, and therefore $a_i$, $b_i$ are independent identically distributed. The rest is all mathematically rigorous. You can forget about $A_i,B_i$ and focus only on $a_i,b_i$.

If your sample size $N$ is large enough, central limit theorem implies that $\sum_{i=1}^N a_i / \sqrt{N}$ is close in distribution to the normal, provided $a_i$ are independent, which is true in your case. Furthermore the boundedness of $a_i$ actually helps you here because you can get uniform bound on the convergence rate, in terms of metrics like Levy's metric or Wasserstein's metric; the rate of convergence depends on first three moments, by the Berry-Esseen theorem. With that you can estimate the probability that $\frac{1}{N} \sum a_i = \bar{a}$ given $\mathbb{E}(a_i) \equiv 0.5$, pretending that $\frac{1}{\sqrt{N}} \sum_{i=1}^N a_i$ follows the normal distribution, with a tiny error term coming from CLT.

Now given that your sample size is pretty small, and very little is known about the distribution of $A_i,B_i$ and therefore $a_i,b_i$, it is always a good idea to err on the side of caution. Let's say the sample mean of $a_i$ you observed was $\bar{a}$. So you could try to compute the following quantity: $$ \sup_{\mathcal{L}(A,B)} \mathbb{P}(\mid \frac{1}{5} \sum_{i=1}^5 a_i - 0.5 \mid > \mid \bar{a} -0.5\mid \big\rvert \mathbb{E}(a_i) \equiv 0.5).$$

$\endgroup$
1
$\begingroup$

You definitely do not want to compare the proportion of A directly to the proportion of B, for the reason that you already stated: these proportions must always add to 1 and therefore are completely redundant with each other.

Instead, what you want to do is compare the proportion of A with 0.5, and determine whether it is significantly less than or greater than 0.5 (or perhaps not significantly different).

The simplest thing to do is to take your five proportions of A (ie, [0.2, 0.3, 0.25, 0.2, 0.15]) and to compare these values versus 0.5 using a one-sample t-test.

You should plot a distribution of your proportions and see if it looks normal. I am concerned that with only n=5 you will not be able to make any strong conclusions. I would guess you will need at least 3x more data to be at all confident. With more data, we could also evaluate the benefits of other quantifications of the effect size, for instance, the log of the ratio B:A instead of the proportions.

If it is not possible to take more data, you need to provide more details of the experiment and there may be something more nuanced you can do to get more statistical power, for instance, assuming a certain distribution for the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.