# confidence intervals around proportion on multilevel data

## I've posted this question another listserv (statalist) and have not received any responses. Any feedback is welcome.

I’m doing an analysis of applications and selections for grants. I do not have a way to identify the unique person in the data (and a person can apply for multiple grants), so any analysis has to be done at the level of the application. I show an example table (made up data). It shows for each year, of all applications, what percentage came from minorities, what percentage came from whites, and what % form race unknown. (note I have an equivalent table of % selections that I would also want to do the equivalent)

I was asked to provide some kind of “margins of error” around the estimates.

I thought about adding confidence intervals around each proportion, which seems to be easily obtained in Stata using proportion gender, over(year) (shows the proportion, std. Err, and logit 95% conf. Interval)

My question is -

1. Is it a good idea to show the confidence intervals around every one of these proportions? What am I getting from this? For example, can I use it to say the % of whites was “statistically significantly higher” in 2017 than in 2011 if the confidence intervals do not overlap?

2. In reading about confidence intervals, the examples I’ve found are based on surveys. Here, I have a population, as it is everyone who applied for the grant. Thus, are these confidence intervals appropriate?

3. Is it a major problem to do be showing confidence intervals on the level of the application and not the unique person?

• You need to give numbers instead of percentages. (It would be enough to give the $n$ that corresponds to 100%. Example: If $n=100,$ then proportion .56 gives 95% CI $(.46,.66).$ if $n = 1000,$ then prop .56 gives 95% CI $(.524, .591).$ Why do you want to show CIs? What conclusion of yours is made more strongly with them? What speculation by readers to you want to facilitate? // Be careful: Non-overlapping CIs may or may not mean two proportions are significantly different. Generally, if you want to know such differences, you need to do tests. May 19 at 20:25
• Thanks – post updated to show total n. I’m not 100% sure why to use the CIs or any "margins of error" here. One question is whether there are statistical significant trends (increases/decreases) over time in the % minority, non minority, etc (which I think you need a test for). The other thought is whether it could be used by the reader to tell immediately whether, e.g., 2017 has a lower percentage of minorities than in 2011 (if the CIS do not overlap). From what you’re saying it seems it can’t be used that way. Thus I’m not sure the benefit. May 19 at 21:05
• Do you mean if they DO overlap they may or may not be significantly different? I was under the impression that if they do not overlap, then they are statistically significantly different. May 19 at 21:23
• In my opinion, trying to do tests by combining confidence intervals is seldom a useful strategy. It can go either way. May 19 at 21:54
• Non-overlapping CIs is not an adequate test of difference. It seems like you're perhaps interested in whether there is trend for applications by group and by time. If you have the counts, then I'd think Poisson regression could be useful. If you stick with proportions, then maybe a Binomial regression. I think it's hard to venture too far into statistical tests, though, because you can't really say whether applications are independent. Depending on other data available, certain methods may be better than others to help understand the data May 19 at 21:56

Comment. Here is one kind of test you could do in R, illustrated with your fictitious data.

Are proportions of proposals submitted by (known) whites changing significantly over time?

You give proportions of whites as p.w and numbers of proposals

p.w = c(23,24,25,23,27,29,32)/100
n = c(9,10,10,12,13,13,16)*100


So counts of proposals by whites are as follows:

w = p.w*n
w
[1] 207 240 250 276 351 377 512


And total proposals per year are:

n
[1]  900 1000 1000 1200 1300 1300 1600


In R, prop.test will test whether all seven proportions are equal. [Continuity correction declined on account of moderately large sample sizes.] Because of the very small P-value, the null hypothesis, that the proportions are the same in all years, is rejected. Then you might do ad hoc tests to see where significant differences lie.

prop.test(w,n,cor=F)

7-sample test for equality of proportions
without continuity correction

data:  w out of n
X-squared = 46.465, df = 6, p-value = 2.392e-08
alternative hypothesis: two.sided
sample estimates:
prop 1 prop 2 prop 3 prop 4 prop 5 prop 6 prop 7
0.23   0.24   0.25   0.23   0.27   0.29   0.32


Another possibility is to make a $$3\times 7$$ contingency table TBL of counts for Minorities, Whites, Unknown by year. Then a chi-squared test for homogeneity would test whether the mix of proposals among the the three categories is changing. (If so, this might be only because the proportions for Unknown are changing for bureaucratic reasons.)

I won't do the test because I don't have real data, but the syntax in R is chisq.test(TBL, cor=F). If you get a significant result, you can explore further with ad hoc tests (making sure to use some method, such as Bonferroni, for avoiding 'false discovery' from repeated tests on the same data).

There are many other kinds of tests that can be done with such data. Mainly, this is just to get you started thinking about what you want to do. Then you might edit your current Question or start a new one, asking about specially targeted methods of analysis.

• wonderful, thank you! May 19 at 22:59