2
$\begingroup$

I'm a student and had a question about statistical analysis.

I'd like to compare post-treatment outcomes between two groups: group A who received a traditional drug (n = 200), and group B who received a newer drug (n = 100). Some post-treatment outcomes are continuous, whereas others are binary (e.g. % developing organ failure, mortality). The question is: which drug is superior? Group allocation was non-random, i.e. based on doctors' choices/clinical factors - more of an observational study than an RCT.

It has been suggested to me that I should code the drug such that Group A is '0' and group B is '1', and run regression models to evaluate the explanatory power of drug choice on the various outcomes (linear for continuous dependent variables, logistic for binary dependent variables).

My understanding so far is that there are 2 advantages of regression: controlling for confounders, and seeing associations of other variables.

However, I also read that there are 5 assumptions for linear regression models. When I began testing my models for these, I found that my models don't have normal residuals (and so homoscedasticity also becomes hard to test). I don't have the time nor skill to transform / manipulate my data. The other issue is that for some outcomes, there are far fewer data points i.e. n = ~40.

My question is: would a t-test/Mann-Whitney U (depending on normality), comparing mean values for the continuous outcomes between Group A and Group B be okay in this scenario, instead of regression? I've done demographic comparisons and only age differs between the groups; can I report the t-test/MWU P-values with the caveat that any significant results could also be due to differences in age? And for binary outcomes like % organ failure, I could just use 2-tailed tests of proportion?

Would be grateful for some help! I think some versions of this question have been asked before, but I couldn't find a conclusive answer.

Improve this question
$\endgroup$
1
  • $\begingroup$ Welcome to Cross Validated! I hope you get a full answer explaining this, but the (equal variance) t-test is equivalent to OLS linear regression on a binary predictor (using something called. “Wald test” of the coefficient), and the Wilcoxon test is equivalent to a proportional odds ordinal logistic regression on a binary predictor (using something called a “Score test” of the coefficient). $\endgroup$
    – Dave
    Commented Apr 24, 2022 at 13:41

2 Answers 2

Reset to default
1
$\begingroup$

The (equal variance) t-test is equivalent to OLS linear regression on a binary predictor (using something called a “Wald test” of the coefficient), and the Wilcoxon test is equivalent to a proportional odds ordinal logistic regression on a binary predictor (using something called a “Score test” of the coefficient). You do not need to separate your thinking into hypothesis testing approaches and regression approaches, since there is an equivalence between the two.

Hypothesis testing and regression often get taught as separate areas of statistics, but you end up unlocking a lot of power when you start seeing standard tests as special cases of regression, as that allows for a lot of accounting for additional variables (such as going from ANOVA with just the groups to ANCOVA that considers the groups AND some additional predictor that you believe influences the outcome).

Improve this answer
$\endgroup$
3
  • $\begingroup$ Hi Dave! Thanks so much :) Couple of follow-up questions: my outcome variables are mostly non-parametric; is the Mann-Whitney U also equivalent to linear regression with a binary predictor like the t-test? Also, what if the groups have different variances? Finally, I now get that regression will be more powerful as it considers other factors, but given that my models don't fulfil the regression assumptions, just for this study would you suggest I proceed with Mann-Whitney U instead? Just thinking about how I'm going to justify my choice of statistical test $\endgroup$
    – ss_99
    Commented Apr 24, 2022 at 16:18
  • $\begingroup$ 1) Wilcoxon is equivalent to the proportional odds ordinal logistic regression, not standard OLS linear regression. 2) Different variances are a good topic for a separate question. 3) Since t-testing and Wilcoxon-testing are regressions, I do not follow what you mean about not satisfying the assumptions of regression and wanting to hypothesis test instead of so regression. // Given the complexities of your problem, your best bet might be to work with a consulting statistician. $\endgroup$
    – Dave
    Commented Apr 25, 2022 at 3:17
  • $\begingroup$ Thanks! Although I now understand that t-tests are special cases of regression, and that a t-test is essentially equivalent to a simple linear regression, I thought t-tests don't require the same assumptions as a multivariate linear regression model, which is why i considered them an alternative for my study. But overall, what are my alternatives to analyse a dataset that doesn't meet the assumptions of regression? (since your answers imply that they won't meet the assumptions of t-tests either)? $\endgroup$
    – ss_99
    Commented Apr 25, 2022 at 13:50
0
$\begingroup$

Would a t-test/Mann-Whitney U (depending on normality), comparing mean values for the continuous outcomes between Group A and Group B be okay in this scenario, instead of regression?

No, because two-sample comparisons tests don't allow for covariate adjustment. You should do regression analysis instead as argued by @Dave.

You are working with a non-randomized study. And since treatment assignment wasn't randomized you cannot assume that groups A and B are comparable (exchangeable). If there is a difference between post-treatment outcomes between A and B, is it because the new treatment made a difference or because patients given the new treatment were younger/older/sicker/healthier/...? So you have to include covariates in the analysis to adjust for any systematic differences between groups. (And even then you have to assume you've adjusted for all necessary covariates, an unverifiable assumption.)

I've done demographic comparisons and only age differs between the groups.

It sounds like you ran multiple separate tests to check if groups A and B differ in terms of a demographic covariate. However, you don't know the power of the individual tests and more importantly doctors determine treatment by considering all (known) relevant factors together. If you want to check for balance between groups A and B you can estimate propensity scores by fitting a model to predict treatment assignment based on all known covariates.

can I report (...) P-values with the caveat that any significant results could also be due to differences in age?

You need a much stronger caveat than that. Even if you include either the demographic covariates or the propensity scores in the regression, you cannot claim that the analysis is adjusted for all systematic differences between groups A and B.

That's the challenge with non-randomized studies. We can adjust only for covariates that have been measured, while what we really want to do is adjust for all covariates (both latent and measured) that differ systematically between groups. In your study, you want adjust for all factors that doctors took into consideration when they decided whether to assign the traditional treatment or the new treatment.

So in the caveat, you should list all covariates which you adjust for as well as mention any additional factors that are considered clinically relevant.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.