1
$\begingroup$

I am conducting an experiment and I am deciding whether to use a forced-choice paradigm (participants choose A or B), or a continuous rating paradigm (rate each stimulus A and B on a Likert scale).

There are a lots of issues to consider, but I am specifically interested in how this choice affects my statistical power to detect an effect.

I know, for instance, that a binomial test could not detect a difference at p<.05 with fewer than six subjects, whereas it's possible that a paired t-test could.

Practically speaking, does a paired t-test usually have higher power than a binomial test (or for that matter, chi-square?)

$\endgroup$
2
$\begingroup$

The simple answer is that all else being equal, if you have a scale and dichotomize it into just A or B, you will generally have less power than if you used the continuous scale (see e.g. Senn, Stephen. "Disappointing dichotomies." Pharmaceutical Statistics 2.4 (2003): 239-240. or Senn, Stephen. "Being efficient about efficacy estimation." Statistics in Biopharmaceutical Research 5.3 (2013): 204-210.). You should of course consider whether somehow all else is not equal. E.g. it could be possible that one particular way of asking a question (e.g. forcing people to make a clear choice) is somehow better at elicitating a clear preference. However, I would like to see some pretty compelling evidence for that before I would abandon a more nuanced outcome in favour of a dichotomized outcome.

$\endgroup$
0
$\begingroup$

Besides the issue if dichotomous vs continuous variables as discussed by @Bjorn, the is the issue of paired vs non-paired tests. The dichotomous correspondent of the paired t-test is not a binomial test (which is unpaired) but the McNemar test - a paired dichotomous test. In my experience, paired tests are more powerful than non-paired tests, but I don't know of any proof of that.

$\endgroup$
  • $\begingroup$ Hmm I'm not sure I agree. Both the binomial and paired t-test are, in my case, "within-subject". And of course, a paired t-test is the same as a single sample t-test on the difference scores. I don't have a 2x2 contingency table, so I couldn't perform a McNemar test. $\endgroup$ – Jeff Mar 2 '16 at 2:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.