2
$\begingroup$

I do a lot of studies in which we have a disease/outcome and then we collect a lot of information on the patients such as age, gender, BMI, comorbidities, lifestyle factors etc. and then we run a model to see if something sticks. These studies are not to be considered as something you would create new guidelines for, but merely to give future researchers ideas.

So far we've done a lot of stepwise-type of model building in which we do a univariate regression for each independent variable and then include those with alpha <= 0.2 in a multiple regression model and finally report those with alpha <= 0.05 in the multiple model.

I've never quite understood this approach as I would normally use multiple regression to select variables of interest that may influence each other, regardless of their presenting alpha. To me it sounds like something you tell a graduate student to give them something to model their data with, without knowing any statistics to make sure they don't overfit their model.

My question is, I'm considering stopping with the stepwise-type of approach and starting to report studies more in the way of selecting which variables I included in a multiple regression model subjectively based on clinical reasoning.

Would this not strengthen the statistical part of the study?

Improve this question
$\endgroup$
2
  • $\begingroup$ The first approach is wrong (inference cannot be done on the subsequent models anymore), see stats.stackexchange.com/questions/20836/… for more information, if you have not seen it already. $\endgroup$
    – user2974951
    Commented Jan 6, 2020 at 9:32
  • 1
    $\begingroup$ Thank you, the reasons listed there are also why I want to move away from the approach. Would my idea of subjectively picking variables to include in a multiple regression be better fitting for such research as stated? $\endgroup$
    – Paze
    Commented Jan 6, 2020 at 9:36

2 Answers 2

Reset to default
2
$\begingroup$

The approach that is currently used is wrong for many reasons, see Algorithms for automatic model selection for a nice summary. As for alternatives, there are two (three) main ones, depending on your goal:

  1. As you suggested, use domain knowledge to filter the data and then run one model, in which case inference holds and can be reported for all variables. Note: you really should use only domain knowledge and not use the data itself to filter the data, as that would just be another form of overfitting. Meaning you should not use summaries and plots to filter the data (some would call it data leakage). Note2: this won't necessarily "strenghten" the statistical model (it may end up being a worse model), but it will make it correct for interpretation.

Alternatively use a train/test set approach, do all the filtering and processing on the train set and then test your model (once) on the test set, and report these results.

  1. Use more modern approaches which incorporate regularization, such as LASSO. Here you can include all (or most) variables and hope the model will filter all the redundant variables for you. The downside is that you won't get inference results, but the model will select "important" variables.
Improve this answer
$\endgroup$
7
  • $\begingroup$ Thank you, this is very informative. Can you explain why regularization models cannot be used for inference? And does this mean they are only used for association? If so, why use a regression model in the first place? $\endgroup$
    – Paze
    Commented Jan 6, 2020 at 10:17
  • $\begingroup$ @Paze Specifically in the case of LASSO, the model works by introducing bias in the model (mainly to prevent overfitting, but also to prevent collinearity). In which case inference is not meaningful anymore, since the coefficients are biased, although the model is still good, and can be considerably better than a similar linear model at making predictions. The selected variables are important for making predictions and so they are related to the outcome, however the effect of the variables is now distorted. If you are interested in the exact effects of the coefficients then you would use a LM. $\endgroup$
    – user2974951
    Commented Jan 6, 2020 at 10:57
  • $\begingroup$ All right, thank you. I'll look into LASSO but to recap it's like a statistical test of association, but for a multiple data set? So it says whether the model as a whole is significant or not. I've tried putting a LASSO model in stata on a set and it doesn't output any coifficients, just a lambda. So it can't predict whether variables are negatively or positively associated and it can't predict which variables are significant? Just whether the entire model is? Maybe I should ask this as a separate question? $\endgroup$
    – Paze
    Commented Jan 6, 2020 at 13:01
  • $\begingroup$ @Paze No, it's not a test and it doesn't measure association. It's a linear model with L1 penalization, so a regularized linear model. It's just like a linear model, but it has something extra in it's loss function, so it doesn't work quite the same. This something extra (lambda parameter) determines the amount of bias. It does output coefficients, so you can determine their effect and sign. All the non-zero variables would be "significant". You must have missed something. $\endgroup$
    – user2974951
    Commented Jan 6, 2020 at 13:09
  • $\begingroup$ Thank you I'll look into it! $\endgroup$
    – Paze
    Commented Jan 6, 2020 at 16:36
0
$\begingroup$

Considering you're comfortable with the idea of making subjective assumptions, and it's not a model that will determine if someone lives or dies, I would suggest a Bayesian model might be a good option.

This would allow you to state your assumptions explicitly and visualize them in a graph, so it can be audited and interpreted in a fairly straightforward way, and still yields quantitative outputs.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.