# Confidence interval for samples from an empirical distribution

I have data where a number of respondents are asked to rate the characteristic of a set of subjects on a scale of 1 to 100. A different number of respondents were asked to rate every subject, so we have differing sample sizes. While the actual dataset has much more observations, an example dataset is:

$$\begin{array}{|c|c|} \hline \textbf{Subject ID} & \textbf{Score} \\ \hline 1 & 72 \\ \hline 1 & 62 \\ \hline 2 & 100 \\ \hline 2 & 55 \\ \hline 2 & 100 \\ \hline 2 & 82 \\ \hline 2 & 72 \\ \hline 3 & 75 \\ \hline 3 & 83 \\ \hline 3 & 100 \\ \hline \end{array}$$

Now what I would like to do is construct confidence intervals for the mean scores of the different subjects, preferably based on the empirical distribution found in the entire sample. We can see that $$\bar{Y}_1=(72+62)/2=67$$, but since sample sizes are different it would be dishonest to only compare means between subjects, confidence intervals would convey more information. What accurate ways are there to construct confidence intervals in a case like this? The standard assumption of underlying normality does not hold, among other things there are large peaks at 100, 75 and 50 in the histogram for 'Score'. If I understand it correctly, I see a couple of practical difficulties in this situation:

(a) Apparently, some of your 'respondents' do not understand the rating system or are refusing to cooperate with it. The presence of many scores of 25, 50, 75, 100 shows that some of them would be more comfortable with a 4-point scale than with a 100-point scale. If the score is supposed to represent something like overall performance, then you might ask respondents to rate subjects on four or five 20 or 25-point sub-scales, and then give the sum of those as the 'score'.

(b) Because you have different respondents and even different numbers of respondents rating each subject, it is not clear that ratings for one subject are comparable with ratings for another. So, depending on how respondents are chosen or volunteer, there may be a difficulty of using this system to compare subjects.

Nonparametric bootstrap CI. However, your precise question seems to be how to find a confidence interval for the population mean $$\mu,$$ given a set of ratings from a population that is not normal--and does not have any other known distribution. You can do that with a 'nonparametric bootstrap' procedure.

Rationale for bootstrap. Suppose you have $$n = 100$$ scores from an unknown population with mean $$\mu$$ and that the observed sample mean is $$\bar X = \bar X_{obs}.$$ In order to get a confidence interval (CI) you need to have some idea how the population mean scores $$\bar X$$ vary about the population mean $$\mu.$$ Let's call that random discrepancy $$D = \bar X - \mu.$$

If you knew something specific about the population you might be able to find values $$L$$ and $$U$$ such that $$P(L < D < U) = P(L < \bar X - \mu Then a 95% CI for $$\mu$$ would be of the form $$(\bar X - U, \bar X - U).$$

"Bootstrap world." Because we do not know $$L$$ and $$U,$$ we enter the bootstrap world to approximate these values. We take a large number $$B$$ of 're-samples' with replacement of size $$n$$ from the sample of $$n$$ observations. We find the sample mean $$\bar X_j^*$$ for the $$j$$th re-sample. Because our best guess at $$\mu$$ is $$\bar X_{obs}$$ we temporarily, use the notation $$\mu^* = \bar X_{obs}.$$ Then for each of the $$B$$ re-samples, we find $$D_j^* = \bar X_i^* - \mu^*.$$ Then we find quantiles .025 and .975 of $$D_1^*, D_2^*,D_3^*, \dots D_B^*,$$ calling the quantiles $$L^*$$ and $$U^*,$$ respectively.

Bootstrap CI. Then, back in the real world, we use $$(\bar X_{obs} - U^*, \bar X_{obs} - L^*)$$ as a 95% nonparametric bootstrap CI for $$\mu.$$

Data for example. For an example, I simulated the suitably 'lumpy' sample of $$n = 100$$ scores shown in the histogram below. Tick marks show positions of exact scores, where there are a few ties (especially around values divisible by 25). R-code for bootstrap CI. The R code below shows how to make a 95% nonparamatric bootstrap CI for $$\mu;$$ the data are in the vector x, its mean is a.obs (67.96) and the suffixes -re are used instead of $$*$$'s to represent re-sampled quantities.

set.seed(924)
B = 100000;  n = length(x);  a.obs = mean(x)
d.re = replicate( B, mean(sample(x,n,rep=T)) - a.obs )
UL.re = quantile(d.re, c(.975, .025))
a.obs - UL.re
97.5%  2.5%
63.09 72.98


Thus a 95% CI for the population mean $$\mu$$ used to simulate the data is $$(63.09, 72.98).$$ The fundamental assumptions of a nonparametric bootstrap CI are that the data are a random sample from a population that has mean $$\mu.$$

Confidence intervals based on the t distribution for non-normal samples this large are known to give useful results, provided skewness is not extreme and there are no extreme outliers. For the 100 observations in x, the t 95% CI $$(62.91, 73.01)$$ is not hugely different. Of course, assuming that the data are normal "adds information" so we can't expect the two CIs to be exactly equal.

Note: The data in x were simulated according to the R program below; the population mean is $$\mu = 68.75,$$ which is covered by the bootstrap CI computed above.

set.seed(1018)
x1 = 100*rbeta(75, 2, 1)
x2=c(rep(25,2), rep(50,6), rep(75,7), rep(100,10))
x3 = x2 + runif(25, -2, 0)
x = c(x1, x3); x=round(x)

• Thank you for your elaborate, extensive answer. I fully agree with your points a) and b), unfortunately this is the data we have, not the data we would like. I suspected nonparametric bootstrapping would be the way to go and your comprehensive explanation confirms my suspicions. – A. Tom Sep 24 '18 at 21:56
• Unfortunately there is another problem present in the data. I worry that the number of observations for each subjects will be too small for accurate bootstrapping. While in your example we have (a very nicely) simulated 100 observations for a subject, in the real dataset we only have 10 on average. Would you have any insights on how to handle this? – A. Tom Sep 24 '18 at 22:04
• Good point. With little to go on, I took my cue from your histogram. // For rating one subject based on 10 respondent scores, you might see if you get useful CIs often provided in computer output along with test result of a one-sample Wilcoxon (signed-rank) test. For comparing two subjects against each other, you might try two-sample Wilcoxon (rank sum test) or another permutation test, but you'll probably need $\ge$ 8 scores for each subject to get useful results. // Post actual data for 1 or 2 'typical' subjects and I'll take a look. – BruceET Sep 24 '18 at 23:48
• However, if about 1/4 of respondents can't do better than score $\pm 25,$ you can't take results about 2 subj judged by small numbers of respondents seriously if avg scores differ only by a few points. That goes whatever method of analysis you use. – BruceET Sep 24 '18 at 23:53
• In R, with data s1 = c(33, 50, 82, 88, 95); s2 = c(25, 30, 45, 53, 70), one sample CI for s1 from wilcox.test(s1, conf.int=T) is $(33, 95)$ for pop. median. CI for difference between subj medians from wilcox.test(s1,s2, conf.int=T) is $(-20, 63).$ Neither result spectacularly useful. Thus the guideline "$\ge 8$" per subj in earlier comment. – BruceET Sep 25 '18 at 18:08