1
$\begingroup$

I thought this would be a relatively straightforward question, but I'm having difficulty finding a decisive answer. Let's say I'm planning to run a study to test the effectiveness of a new type of pain medicine with 100 participants, who will be randomly divided into an experimental group and a placebo group with 50 participants each. I will measure the 50 individuals in each group on a Likert-type pain scale at the beginning of the study (pretest) and again after taking either the medication or a placebo for a month (posttest). What would be the most appropriate statistical test to use to test if there is a significant difference between the impact of the medication vs. the placebo for the two groups (i.e., if the medication was effective at reducing pain)?

Many thanks!

$\endgroup$
1
$\begingroup$

I'll say that, if the hypothesis is that the drug is more effective than the placebo, for instance, that it'll reduce the score in the Likert scale significantly more. You could first transform the Pre and Post medication score, into a single score which would be the difference between the two. Thus a significant difference between the two groups, with this score, would mean a significant improvement of the feeling of pain.

So let's say that the scale is from 1 to 10, with 10 meaning having a lot of pain and 1 having none. You would like to have a significant reduction of the pain, thereby, a negative difference when computing the Post-test score minus the Pre-test score. Plus you would like that difference to be inferior compared to the placebo (i.e. that the people rate their pain as less important with medication than with placebo).

So with this in mind, a one-tailed independent t-test, should do the job. If the assumptions of the t-test aren't met, you could still use a Mann-Whitney U, or apply some transform to the data.

$\endgroup$
1
$\begingroup$

One good idea could be to look up recent work for Phase 3 registration trials (which get a lot of scrutiny by regulators such as the US FDA and get planned with a lot of attention to detail by the pharmaceutical companies running the studies) that used the particular endpoint (or very similar ones).

A very simple model could be a linear regression model for the numeric post-treatment score with treatment as a factor in the model and the numeric baseline score as a covariate. The biggest difficulties with this could be:

  1. Some assumptions for linear models might be so badly violated that it matters (many small deviations don't really matter). E.g. the regression residuals might be very non-normal or the errors might be larger in the middle of the Likert scale than at the ends etc. Ideally, one knows from previous studies whether a linear regression approach is okay (it's always a bit problematic to test assumptions in the study one is analyzing) and it might even be that people have done simulation studies with realistic settings for the exact outcome scale you are interested in.
  2. Even if the model fits quite well, what do you do about those that do not have a score at the end. This is a huge topic in itself and almost independent of the choice of model (see e.g. this guidance). Options include that we assume people are similar to other people with similar covariate values (including that the treatment continued for them even though we did not get to see the final outcome - or perhaps we want to know what would have hypothetically happened if they had finished their treatment) and we could then e.g. do multiple imputation (i.e. you impute, say, 100 datasets and then analyze each, then you aggregate the results - this is nicely handled by standard software). If it's that those that stop treatment would then be like similar people in the control group, then approaches like "jump-to-reference" could be an option.

An alternative is to model the actual ordinal outcome. Frank Harrell has recently talked a lot about this (e.g. here) and has cited some papers on this (such as this one and there's also this). The main advantage is that it attempts to model the actual data generating process and allows adjustment for the baseline score. You can have a look around the data methods forum that Frank Harrell champions and the discussions tagged as "ordinal" on there. That might be a really good place to post a question on how to implement such an ordinal analysis in case you do not find all your questions answered, yet. Point (2) regarding the simple linear model of course still does not go away and needs to be dealt with.

Further alternatives that are less popular include

  • Rank-based methods, but in their basic version they do not make it easy to adjust for the pre-test score. Forming the difference of pre- vs. post is also not that satisfactory, because a change of +2 on a 10 points scale is possible from a baseline of 5, but not from a baseline of 9 (and even when it's possible may mean very different things), an ordinal model can reflect that a lot better.
  • Simple dichotomization (aka "responder analysis") of the outcome (e.g. >5 vs. <=5, or improved by 2 points or achieved score of 9 or 10 etc.), where one could group those that don't finish treatment with those that did not achieve a good outcome (composite estimand strategy). Then a logistic regression with the baseline socre as a covariate could be used. However, this is usually a terrible idea, because you throw away a lot of information by reducing all information to a yes vs. no.
$\endgroup$
2
  • $\begingroup$ Great answer. Quick question that comes from this - when you use a proportional odds model for an ordinal outcome do you get the same efficiency gains from baseline adjustment that you get in linear regression? Presumably no? $\endgroup$ – Lachlan Jun 14 at 12:27
  • $\begingroup$ I don't know for sure (simulate & try it out...), but my guess is that: 1) because the scale is ordinal, there is just less information in the baseline than for a truly continuous baseline (i.e. this does not change whether the analysis model is a linear regression or an ordinal model, it's just about what data you have) 2) how much adjustment helps depends on how well the specified model approximates the drug data generating process - e.g. if you manage to truly write down the "true" DGP with an ordinal model conditional on the baseline, a linear model should not in some sense do better. $\endgroup$ – Björn Jun 14 at 12:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.