How was "above chance" defined in this paper and confidence intervals calculated? [Rating method: Best-worst scaling]

Question

I was reading a paper about ratings (Hollis & Westbury, 2018), precisely, best worst scaling, where in each trial out of N items (in this case N = 4) the best and the worst item have to be selected by participants (on a predefined dimension e.g. the most and least appealing). I was wondering, how they made use of confidence intervals to determine if a participant's responses (compared to other participants' responses) were above chance. In the paper it says: 'If a participant was making choices randomly, their “best” and “worst” choices should be consistent with expectations 50 % of the time. Given that each participant completed 260 trials, the 95 % confidence interval for random guessing would be ± 6.54 %. Thus, we marked any participant with a compliance rate less than 56.54 % as “noncompliant.” 'Can somebody deduce for me how they calculated 6.54% (based on 260 trials with seemingly no standard deviation)? I feel like I'm missing something obvious and would very much appreciate any possible explanations. Further, I'd gladly learn any alternative approaches to taggle the issue of what deviation from the average responses (especially using this BWS rating method) could be considered as above chance.

Answer 1

I must admit that I even do not understand the first assertion: if there are four items, the probability that a random choice of the best item is correct is 25%, not 50%, and the probability that both best and worst are correct is even lower. Maybe there is more information in the reference (Hollis, 2017), which is behind a paywall. Apparently, the authors do not seek a confidence interval for the BWS, but for the "compliance rate", which is a proportion. binom.confint(260/2, 260) returnes a confidence interval $50\%\pm 6.23\%$, which is close to the alleged value.

To obtain a confidence interval for BWS itself, the most straightforward method is a non-parametric approach via the "bias corrected accelerated (BCa) bootstrap". The idea is to randomly draw answers with replacement and compute percentiles of the resulting BWS's (BCa adds some vodoo to reduce bias and impreve asymptotic convergence to the nominal coverage probability). Moreover, it is also possible to estimate the variance of the BWS with the "Jackknife Method".

For details on applying the bootstrap for computing confidence intervals for arbitrary estimators with R sample code, section 6 of this technical report might be helpful.