Confidence Intervals for Likert Items

Question

Background

As part of a community survey looking into environmental habits, I have 22 Likert Scale questions each with six Likert items. As for the sample size, we have over 65 responses to each question.

I am familiar with the analysis of Likert Scales and the various limitations. However I am wanting to look at this differently (even if just for research curiosity). I want to ascribe the confidence interval for each item across each 22 Likert scale questions.

From what I have reviewed, the classical Wald interval for a binomial response is unreliable when the proportion approaches zero and one. So I have discarded that option and that seems to leave the following two.

Wilson score interval

$\hat p_w \approx \frac {n_i +\frac{1}{2}z^2}{n+z^2} \pm \frac {z}{n+z^2} \sqrt{\frac{n_i n_j}{n} +\frac {z^2}{4}}$

Simultaneous Confidence Intervals for Multinomial Proportions

$\hat p_g \approx \frac {n_i +\frac{\beta}{2}}{n+\beta} \pm \sqrt{\frac{\frac{\beta^2}{4}+\beta n_i(1-\frac{n_i}{n})}{n+\beta^2}}$

$n_i$ = number of respondents choosing the $i^{th}$ item

$n_j$ = number of respondents not choosing the $i^{th}$ item

$n = \sum_{i=1}^k n_i$

$k$ = the count of $i^{th}$ Likert items per scale

$\beta$ = the upper $(\frac{\alpha}{k})$ 100th percentile of the $\chi^2$ distribution with 1 degree of freedom

Both $\hat p_w$ and $\hat p_g$ re-estimate the proportion hence the confidence intervals that are not symmetric to the original proportion.

Using some survey data for the above options, we have (using $z$ = 1.96 and $\alpha$ = 0.05).

$$\begin{array}{c|c|c|c|c|c|c} \text{Likert Item} & \text{Count} & p & \hat p_w & \pm CI_w & \hat p_g & \pm CI_g \\ \hline \text{1} & 5 & 0.071 & 0.094 & 0.063 & 0.110 & 0.087\\ \hline \text{2} & 3 & 0.043 & 0.067 & 0.052 & 0.084 & 0.074\\ \hline \text{3} & 15 & 0.214 & 0.229 & 0.095 & 0.240 & 0.126\\ \hline \text{4} & 32 & 0.457 & 0.459 & 0.114 & 0.464 & 0.150\\ \hline \text{5} & 8 & 0.114 & 0.134 & 0.075 & 0.149 & 0.102\\ \hline \text{6} & 7 & 0.100 & 0.121 & 0.072 & 0.136 & 0.097\\ \hline \end{array}$$

Assuming my calculations are correct, two observations:

1. $\hat p_w$ is less than $\hat p_g$.
2. $\hat p_w$ has a smaller confidence interval than $\hat p_g$.

Question

Which is the 'correct' confidence interval to use for a Likert Scale question?

Answer 1

The Wilson confidence interval is superior. Among other places it is implemented in the R Hmisc package binconf function.

But this whole discussion has focused on binary variables which are coarsened versions of Likert ordinal scales. Even though Likert scales are not really interval-valued it is often best to summarize them with a single mean. Then the job can be converted to one of selecting the best performing bootstrap confidence interval approach to get confidence intervals for population means. "Best performing" means the method that has the two tail non-coverage probabilities closest to 0.025 if computing a 0.95 confidence interval.

You can also get estimated means from ordinal regression as discussed here and here.

Answer 2

I do not know the Wilson approach so will not comment this one.

The multinomial approach would be the best approach if and only if the total sample size (here 70) was a universal constant. Indeed, this distribution assumes that the total count (here 70) is a constant. However, if --in taking a new sample-- the total sample might no longer be 70, then this is not the correct approach.

Let me document two alternative approaches.

1) The Clopper-Pearson method This approach, based on Clopper and Pearson (1934) is the exact distribution of a proportion (in isolation). Leemis and Trivedi, 1996, provided an easier computation, that is

\begin{equation} CI_{low} = \left( 1+ \frac{n-x+1}{x F_{2x,\, 2(n-x+1)}((1-\gamma)/2) } \right)^{-1} \end{equation} and \begin{equation} CI_{high} = \left( 1+ \frac{n-x}{(x+1) F_{2(x+1),\, 2(n-x)}(1-(1-\gamma)/2)} \right)^{-1} \end{equation} in which $x$ is the count, $n$ is the total count, and $F$ is the quantile taken from a F distribution with numerator and denominator degrees of freedom respectively, for a $\gamma$ level confidence interval (e.g., $\gamma = .95$).

2) The transformation approach Proportions can be transformed using the Anscombe arcsine transform. When this is done, the resulting proportion has a sampling distribution which is normal and a standard error which is constant at $\sqrt{1 / (4(n+1/2)) } $ (well, sort of a constant as it depends on sample size).

The transformation is $\text{arcsine} \left( \sqrt{\frac{x+3/8}{n+3/4} } \right)$

Thus, the transformed scores have a confidence interval of 1.96 the standard error, i.e., \begin{equation} CI_{low} = \text{arcsine} \left( \sqrt{\frac{x+3/8}{n+3/4} } \right) - 1.96 \sqrt{\frac{1}{4(n+1/2)}} \end{equation} then the scores can be un-transformed, e.g., $$\text{Untransformed }CI_{low} = \frac{(n+3/4) \sin(CI_{low})^2 - 3/8}{n}$$

Note that in either case, the confidence interval limits are not symmetrical.

In your example, we get the following 95% confidence intervals.

Likert item	Count(x)	Total(n)	$p$	CP-$CI_{low}$	CP-$CI_{high}$	Tr-$CI_{low}$	Tr-$CI_{high}$
1	5	70	0.071	0.024	0.159	0.021	0.145
2	3	70	0.043	0.009	0.120	0.005	0.105
3	15	70	0.215	0.125	0.329	0.126	0.318
4	32	70	0.457	0.337	0.581	0.342	0.575
5	8	70	0.114	0.051	0.213	0.049	0.200
6	7	70	0.100	0.041	0.195	0.039	0.182

CP: Clopper and Pearson; Tr: Anscombe transformation.

The advantage of the transform approach is that it can be used to test comparisons and perform ANOVAs when there are multiple proportions to compare.