I am running a backward-selected multiple linear regression to correlate a continuous dependent variable (mussel density) with 10 categorical independent variables (substrate, side of bay, animal presence, etc). After backward selection I end up with a model with an adjusted r^2 of 0.522 that has included 5 out of the 10 independent variables. My question is how to interpret the rest of the data provided by the model. Do the p-values still matter after backward selection (i.e. for the substrate variable some of the substrates have a significant p value and other do not, but the whole substrate variable was included in the model)? Which variables should I report when sharing results of the model? Any help is much appreciated, still new to this!
-
1$\begingroup$ What do you want to do with the model? $\endgroup$– DaveJan 20, 2021 at 23:17
-
$\begingroup$ Did you hold out a random subset of the data for testing your model? That's about the only way you can approximate anything like a valid p-value or $R^2.$ $\endgroup$– whuber ♦Jan 20, 2021 at 23:36
-
$\begingroup$ The goal for the model is to correlate the impacts of these 10 factors on mussel density. So I've got ~200 different sections of shoreline that I've taken all these variables for and now want to see which factors correlate with increased mussel density. $\endgroup$– Joshua HartJan 21, 2021 at 0:43
1 Answer
Stepwise selection as a form of model development isn't well regarded these days. Part of the reason is that it tends to produce 'final' models which explain a great deal of the variance in your sample, but do very poorly when tested against new samples. The p values and R2 statistics from your final model are also invalid, as they ignore the model development process and presume that the final model was specified 'a priori'. Thus, any uncertainty about whether the final model is the 'correct' model (and there's a good chance it isn't) is ignored.
People often say when talking about stepwise methods, don't let your computer think for you. You have collected data for 10 variables you presume may be important for explaining your outcome. Why not present the results of your multivariable regression including all 10 predictors? Then you can summarise each with an effect size (regression coefficient) and the uncertainty around that effect size (e.g. 95% confidence interval). This will tell you plainly which variables are the most important in determining Mussel density, after accounting for the other variables.
If you're worried about overfitting, or want to find the most parsimonious possible model (i.e. most variance explained with fewest possible predictors), a more principled approach might be a shrinkage method like the LASSO. Shrinkage methods and the perils of stepwise selection in the ecology context are explained well in this short letter. The LASSO can be implemented quite simply in the R package GLMNET and there are many good tutorials online that run through the process.
