I have a list of categories A, B, C, etc, each with a probability:

A   .015
B   .005
C   .02
D    ... etc. 

I need to find if the difference between two of these categories, say

diff(C,A) = .02 - .015 = 0.005

is statistically significant. I have been using a Standard Error of Differences with the probabilities but since I don't have a normal distribution (and obviously not even continuous), what test would be recommended?


The source of the data is essentially a list of the categories, these are not experiments I am performing but data that is given to me. The size of the list can vary but let's assume it's reasonable small (under 500).


Accepted the answer below but you should read the comments attached to the answer for full understanding of the resolution.


You left out some pieces that need to be assumed. First of all what is your data to get the estimates of thee proportions? Do you have an experiment where in each trial a category occurs and you have a sequence of trials? If so the probabilities for the list of categories will sum to 1. How many trials n did you run to get the estimates?

For purposes of supplying an intelligent answer that could be what you are looking for I will make the assumption that you have k categories and look at the cases where (1) n is very large and (2) n is fairly small.

In either case under my assumptions the joint probability distribution for number of occurrences of each category in n trials is multinomial. There are k-1 parameters (category probabilities) since you have the constraint that the sum of all the category probailities equals 1. So for example the last category has its probability determined given the probailities fro the othe k-1 categories.

Now category i has probability denoted by p$-i$ and keep in mind that p$_k$ =1 - sum of all the other p$_i$s. Suppose you want to compare category 1 with category 2. The number of cases in category 1 is marginally binomial with parameters n and p$_1$. Similarly the number of cases in category 2 is marginally binomial with parameters n and p$_2$. In case (1) when n is very large you can approximate the difference in the two proportions by a normal distribution. The trick though is that even though you can estimate the variance of the estimates of p$_1$ and p$_2$ the variance for the difference is not the sum of the variances. The estimates are negatively correlated and hence their covariance is negative. For the variance of the difference you need to subtract 2 times the covariance of the estimates. Since the covariance of the estimates is negative this means the variance will be larger than what you would get in the case of independent sets of trials.

The joint distribution of the number in category 1, the number in category two and the number in neither 1 nor 2 (n - the sum of the other two) is trinomial. It's distribution depends on p$_1$ and p$_2$ and the covariance can be expressed as a function of p$_1$ and p$_2$. Those probabilties are estimated as # of category occurrences/n. So we can estimate the covariance by plugging these estimates into the function. Then we can estimate the variance of the difference and normal to have variance approximately 1. Under the null hypothesis that p$_1$ = p$_2$ the normalized statistic will have mean 0 and will be approximately a standard normal. This distirbution can then be used for the test.

In case (2) things are more complicated because the normal approximation may not be good. In that case an exact test must be performed based on the exact trinomial distribution mentiioned earlier. This wikipedia reference gives all the details you need regarding the multinomial distirbution including the covariance which is -n p$_1$ p$_2$ for p$_1$ and p$_2$ that is need for the standard error of the difference estimate.

  • $\begingroup$ Thanks for your response, @MichaelChernick. As noted above, the categories are probabilities, therefore, they sum to 1. In terms of your answer, what test exactly should be used? Is the Standard Error of Differences with probabilities a reasonable one or does it depend highly on the size of n. It's a bit unclear what your suggestion is, though it is informative in thinking about the distributions. $\endgroup$ Sep 13 '12 at 20:49
  • 2
    $\begingroup$ @Lillian Yes you use the standard error of the difference but because of the negative correlation it is a more complicated calculation and is not the same as what you would get from comparing two independent binomial random variables. $\endgroup$ Sep 13 '12 at 20:55
  • $\begingroup$ any chance you can update your answer to include the last point you made? It's really informative and may help other people looking through this answer. I will accept your answer after that update. Thank you! $\endgroup$ Sep 14 '12 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.