I was reading a paper about ratings (Hollis & Westbury, 2018), precisely, best worst scaling, where in each trial out of N items (in this case N = 4) the best and the worst item have to be selected by participants (on a predefined dimension e.g. the most and least appealing). I was wondering, how they made use of confidence intervals to determine if a participant's responses (compared to other participants' responses) were above chance. In the paper it says: 'If a participant was making choices randomly, their “best” and “worst” choices should be consistent with expectations 50 % of the time. Given that each participant completed 260 trials, the 95 % confidence interval for random guessing would be ± 6.54 %. Thus, we marked any participant with a compliance rate less than 56.54 % as “noncompliant.” 'Can somebody deduce for me how they calculated 6.54% (based on 260 trials with seemingly no standard deviation)? I feel like I'm missing something obvious and would very much appreciate any possible explanations. Further, I'd gladly learn any alternative approaches to taggle the issue of what deviation from the average responses (especially using this BWS rating method) could be considered as above chance.
How was "above chance" defined in this paper and confidence intervals calculated? [Rating method: Best-worst scaling]
I must admit that I even do not understand the first assertion: if there are four items, the probability that a random choice of the best item is correct is 25%, not 50%, and the probability that both best and worst are correct is even lower. Maybe there is more information in the reference (Hollis, 2017), which is behind a paywall. Apparently, the authors do not seek a confidence interval for the BWS, but for the "compliance rate", which is a proportion.
binom.confint(260/2, 260) returnes a confidence interval $50\%\pm 6.23\%$, which is close to the alleged value.
To obtain a confidence interval for BWS itself, the most straightforward method is a non-parametric approach via the "bias corrected accelerated (BCa) bootstrap". The idea is to randomly draw answers with replacement and compute percentiles of the resulting BWS's (BCa adds some vodoo to reduce bias and impreve asymptotic convergence to the nominal coverage probability). Moreover, it is also possible to estimate the variance of the BWS with the "Jackknife Method".
For details on applying the bootstrap for computing confidence intervals for arbitrary estimators with R sample code, section 6 of this technical report might be helpful.
$\begingroup$ Thank you so much for your helpful response with R commands for binomial confindence intervals that may have been used in this paper and the additional link on how to bootstrap confidence intervals. And I'm sorry for not realising, that the DOI doesn't lead to the free pdf of the paper, I changed the link in my original post now. $\endgroup$– MissNyuuJun 7, 2022 at 15:50
$\begingroup$ @MissNyuu The original link already was accessible for some reason (but the self-archived version is presumably a safer bet for accessibility in the future). I was referring to the reference "Hollis (2017)" in that paper, which is cited as a justification of the numbers. Maybe the reasoning is detailed there. $\endgroup$– cdalitzJun 7, 2022 at 16:17