# prop.test: correcting for large number of observations

This is not really a coding question but more of a statistical question.

I'm doing a proportions test on multiple proportions for many subjects.

For example, subject 1 will have multiple proportions (multiple "successes per total trials"), and subject 2 will have multiple proportions. And for each subject we're testing if all these proportions are the same. For each subject, there are multiple proportions where there is number of successes per total trials. The proportions could range from being 30 successes out of 60 to like 300 successes out of a 1000 (just to show the range of trials and successes for each subject). Furthermore, for each subject, there could be varying number of proportions. Subject 1 could have 50 proportions, whereas subject 2 could only have 2. The idea is that we're trying to test that for each subject that all of their proportions are the same, and then reject if they are different.

However, I'm realizing that subjects that have many more proportions, will have more significant p-values than subjects that only have 2 proportions, when using the prop.test. I was wondering if there is a way to approach this problem in a different way. Any sort of correction I could do, or take into account the number of positions.

Any suggestions would be helpfil.

• The issue might best be thought of in terms of power rather than p-values, which are not predictable. The power to discriminate differences increases with the numbers of trials and decreases with the numbers of proportions within each subject. What is unclear to me is why this is a problem. Are you perhaps asking how to conduct the tests properly? If so, exactly how are you currently doing them? – whuber Jul 22 '14 at 16:10
• Thank you for your input. I don't really want the number of proportions to affect the p-value. I don't want to "consider" a subject, just because they have more proportions within them, so therefore more opportunity for them to rather be significant. I'd like to nullify this. As of now, I'm just using prop.test in R, and yes I would like to know how to conduct these tests properly. Thank you for your input! – user3799576 Jul 22 '14 at 16:50
• prop.test already handles this. In all cases a p-value has the same meaning and the same interpretation: Under the null hypothesis it will have a uniform distribution between $0$ and $1$ (subject to some technical limitations due to the discrete nature of the data and inaccuracy of the $\chi^2$ approximation). Under the alternative hypothesis its distribution will be concentrated near $0$. How concentrated depends on how many proportions and how many trials are involved, but the uniformity of the null distribution does not depend on these numbers (again apart from technical limitations). – whuber Jul 22 '14 at 17:15

It's not 100% clear, but if the null for an individual is that all their sample proportions come from a binomial distribution with a constant $p$, you could test that against the alternative that they come from some mixture of $p_t$'s (one for each time) that lead to a different variance.

In effect, you would test for over (or perhaps under-) dispersion. This could be fairly readily done in the context of a GLM.

If you wanted to do it for all subjects at once (they all have a single $p_i$ against the alternative of some mix of proportions, $p_{i,t}$), that might again be done with a GLM as well (with a person-factor).

That assumes you want to make a conclusion about this specific set of people (are their proportions constant or not?).

If instead you want to treat them as a random sample from some population of people and make a conclusion about that population, you may be better off thinking of this in terms of random-effects.

• So, it's not that all subjects have the same proportion, but that within each subject, the proportions are equal. Would this approach work for that? – user3799576 Jul 23 '14 at 17:02
• Sorry, I'm not 100% clear here. You're saying that you're assuming proportions are constant within subject and are interested in testing for differences in proportion across subjects? – Glen_b -Reinstate Monica Jul 23 '14 at 18:30
• We want to test that proportions within a subject are equal. We are already assuming (and are okay with) differences in proportions between subjects – user3799576 Jul 23 '14 at 21:02
• Okay, good that's what my answer discusses - testing the null that "within each subject, the proportions are equal" against alternatives that would change the variance from that; it would rely on the independence assumption however, which may be suspect (if these are over time or along a spatial dimension, you'd want to check for serial dependence in particular). – Glen_b -Reinstate Monica Jul 23 '14 at 22:26