When do I need something "fancier" than multiple regression?

Question

Let me describe my data, without getting TOO specific.

I have ~1000 words. They have all been rated on some dimension (i.e., "DV"). There are 30 properties that can either be present or absent in the words (i.e., "X1":"X30").

I want to know if any properties are more common in words that are higher (or lower) on the DV.

I've taken a few swings at answering this. I have run an adaptive LASSO. I've also run a random forests model with boruta feature selection.

I was told that these two approaches may be good for prediction but not so good for explanation. My main objective isn't to predict new data, but to understand which of my 30 properties are related to the DV.

So I just ran a plain old multiple regression. The residuals are all distributed nicely. The VIF for each predictor is also < 3.

So, is this enough? For some reason, I've gotten it in my head that it isn't, but now that I think about it, I can't explain why. I think my worry was that it is too "liberal" and that with a large enough sample some predictors are bound to be significant.

So my questions are:

1. Is there any reason that this multiple regression isn't enough?

2. Are there ways to "augment" it if not? I've thought about stepwise regression, but in all honesty I'm not sure why I would need that if there is no multicolinearity.

Edit: I will also add the detail that 69/703 pairwise correlations among my predictors are significant at p < .01.

Answer 1

For my money, if your goal is to understand the relationship between your predictors and the outcome, multiple regression is absolutely fine here, BUT you need to worry a bit about multiple comparisons.

You have lots of predictors. Even if none of your predictors are really related to the outcome, just by chance you would expect ~5% of them to come out as significantly associated with it in your data, using multiple regression or any maximum-likelihood method.

LASSO and related methods deal with this by finding a set of regression weights that fit your data reasonably well, while trying to keep the weights small, and having as many weights of $0$ as possible. This is great for prediction, but, for reasons I won't cover here, you can't really interpret the weights estimated using LASSO, particularly when your predictors are correlated.

An alternative way to deal with this is just to use multiple regression, identify all of the significant predictors ($p < .05$), and then, to control your false discovery rate, use something like the Benjamni-Hochberg procedure to throw out the predictors whose effects are too weak.

PS: It's worth mentioning that depending on how many observations and how many features you have, there comes a point where multiple regression no longer works, even with correction for multiple comparisons. I think $\frac{1000}{30} \approx 30$ observations per predictor is probably still fine.

Answer 2

To @Eoin's point, the apparent variable importance is a very unstable quantity when the sample size is not in the millions. What exposes the difficulty of the task and provides actionable information is to use the bootstrap to get confidence intervals on importance ranks of all the predictors simultaneously. The more predictors you have the more difficult it is to select strong predictors from them, and the wider will be the rank confidence intervals. Likewise when you have collinearity. The ranks of importance can be computed on any measure including univariate correlations, partial $R^2$ in a multiple regression model, $\chi^2$ when using maximum likelihood, etc. An example with R code may be found in Section 5.4 of the RMS course notes.

Answer 3

1. It's always a balance trying to balance predictive ability and interpretation. You can try to use LASSO or other shrinkage methods if you would like to emphasize prediction a bit more than multiple regression.
   This may improve predictive ability while preserving some level of interpretability. If you transformed your data, I believe there are interpretability issues with shrinkage methods. Of course, if you want to perform inference on the parameters (t-test/F-test), you will not be able to do this with shrinkage methods (as far as I know).

2. You can still do some variable selection analysis even if collinearity is NOT an issue. You may find, for example, that a smaller model may have a better/equivalent AIC or Adjusted $R^2$ than the full model.

Answer 4

Something fancier than a univariable or multiple regression model is needed when there is a very clear, very complicated non-linear relationship between the endpoint and covariates that cannot be addressed using a routine link function or transformed covariate. If your paneled scatter plots show clouds of data points with suggestive trends then anything fancier could be considered overfitting.

Based on your labeling of what is the dependent and independent variables, your model will tell you if any DV's (high or low) are more common in words with certain properties. Your description had this reversed, suggesting reversal of dependent and independent variables.

Yes, with a large enough sample some of your predictors are bound to be statistically significant, and that is a good thing. In fact, it is a great thing. Don't be disappointed with small p-values. However, just because you have the power to detect an effect does not imply it is a meaningful effect to discuss and report.

As decaf and Eoin have suggested, you can use LASSO and FDR methods among others to weed out independent variables. My preference is to use the p-value as a continuous index for the weight of the evidence by running univariable models and sorting the predictors by ascending p-value, as well as by the magnitude of the estimated effect size. Those predictors with the smallest p-values do the best job explaining your endpoint for your particular data set and stand the best chance of re-demonstrating their effects in a repeated experiment. If this "short list" is still a bit too long you can choose to highlight only the top, say, $k$ independent variables on the list. If we think in terms of Neyman-Pearson error rates instead of evidential p-values, we can talk in terms of per-comparison error rates so that no adjustments to the p-values are necessary, acknowledging that family-wise error rates may very well be higher. The key is to interpret the results as tentative evidence, not irrefutable proof. Those independent variables with the largest effect sizes are the most interesting to discuss and report, even if the p-value does not reach a conventional cutoff like 0.05.

The independent variables that were promising in univariable models can be explored together in a multivariable model to see if the conditional effects persist in smaller subgroups. I suggest creating paneled and grouped plots to explore possible interaction effects. However, this can become very nebulous very quickly. For that reason I prefer to keep emphasis on the univariable analyses that describe the higher-level conditional effect for each independent variable. This may seem elementary, but it may be naive to slice and dice the data by a dozen or more independent variables simultaneously and discuss a miniscule effect in an obscure subgroup especially when the results are tentative evidence and not irrefutable proof. It may be worthwhile to define a subpopulation and fit a simpler model to the data available on this subpopulation, rather than trying to interpret a multivariable regression model. Such a stratified model may provide a better fit to the data and the reduced sample size will produce more conservative inference.

No matter what modeling technique you choose the only true validation is to repeat the experiment and see if the same model does well at explaining the dependent variable. If the experiment can be replicated many times then what was tentative evidence, taken together, becomes irrefutable evidence.