4
$\begingroup$

Let me describe my data, without getting TOO specific.

I have ~1000 words. They have all been rated on some dimension (i.e., "DV"). There are 30 properties that can either be present or absent in the words (i.e., "X1":"X30").

I want to know if any properties are more common in words that are higher (or lower) on the DV.

I've taken a few swings at answering this. I have run an adaptive LASSO. I've also run a random forests model with boruta feature selection.

I was told that these two approaches may be good for prediction but not so good for explanation. My main objective isn't to predict new data, but to understand which of my 30 properties are related to the DV.

So I just ran a plain old multiple regression. The residuals are all distributed nicely. The VIF for each predictor is also < 3.

So, is this enough? For some reason, I've gotten it in my head that it isn't, but now that I think about it, I can't explain why. I think my worry was that it is too "liberal" and that with a large enough sample some predictors are bound to be significant.

So my questions are:

  1. Is there any reason that this multiple regression isn't enough?

  2. Are there ways to "augment" it if not? I've thought about stepwise regression, but in all honesty I'm not sure why I would need that if there is no multicolinearity.

Edit: I will also add the detail that 69/703 pairwise correlations among my predictors are significant at p < .01.

$\endgroup$

4 Answers 4

6
+50
$\begingroup$

For my money, if your goal is to understand the relationship between your predictors and the outcome, multiple regression is absolutely fine here, BUT you need to worry a bit about multiple comparisons.

You have lots of predictors. Even if none of your predictors are really related to the outcome, just by chance you would expect ~5% of them to come out as significantly associated with it in your data, using multiple regression or any maximum-likelihood method.

LASSO and related methods deal with this by finding a set of regression weights that fit your data reasonably well, while trying to keep the weights small, and having as many weights of $0$ as possible. This is great for prediction, but, for reasons I won't cover here, you can't really interpret the weights estimated using LASSO, particularly when your predictors are correlated.

An alternative way to deal with this is just to use multiple regression, identify all of the significant predictors ($p < .05$), and then, to control your false discovery rate, use something like the Benjamni-Hochberg procedure to throw out the predictors whose effects are too weak.

PS: It's worth mentioning that depending on how many observations and how many features you have, there comes a point where multiple regression no longer works, even with correction for multiple comparisons. I think $\frac{1000}{30} \approx 30$ observations per predictor is probably still fine.

$\endgroup$
2
  • $\begingroup$ Thanks very much! Is the reason that LASSO coefficients aren't interpretable to do with the way that it deals with correlated predictors (essentially choosing one)? Also, when you say that at a certain point MR doesn't work any more, is that because of the FDR with so many data points? $\endgroup$
    – Dave
    Nov 15, 2021 at 18:08
  • $\begingroup$ a) Yes, exactly. b) It's simpler than that: the more irrelevant predictors you have, the more noise there is, so you'll need more data to compensate. $\endgroup$
    – Eoin
    Nov 16, 2021 at 21:35
2
$\begingroup$

To @Eoin's point, the apparent variable importance is a very unstable quantity when the sample size is not in the millions. What exposes the difficulty of the task and provides actionable information is to use the bootstrap to get confidence intervals on importance ranks of all the predictors simultaneously. The more predictors you have the more difficult it is to select strong predictors from them, and the wider will be the rank confidence intervals. Likewise when you have collinearity. The ranks of importance can be computed on any measure including univariate correlations, partial $R^2$ in a multiple regression model, $\chi^2$ when using maximum likelihood, etc. An example with R code may be found in Section 5.4 of the RMS course notes.

$\endgroup$
3
  • $\begingroup$ Thanks for this. My impression is that classifying predictors as relevant versus irrelevant should be more stable than ranking predictors in order of relevance, but I don't have evidence to back this up. Would you have any pointers? $\endgroup$
    – Eoin
    Nov 17, 2021 at 14:22
  • 1
    $\begingroup$ No this is precisely what doesn't happen. Being on the right side of an arbitrary boundary for relevance is highly unstable. $\endgroup$ Nov 17, 2021 at 16:47
  • $\begingroup$ Interesting, thanks. $\endgroup$
    – Eoin
    Nov 19, 2021 at 12:25
1
$\begingroup$
  1. It's always a balance trying to balance predictive ability and interpretation. You can try to use LASSO or other shrinkage methods if you would like to emphasize prediction a bit more than multiple regression.
    This may improve predictive ability while preserving some level of interpretability. If you transformed your data, I believe there are interpretability issues with shrinkage methods. Of course, if you want to perform inference on the parameters (t-test/F-test), you will not be able to do this with shrinkage methods (as far as I know).

  2. You can still do some variable selection analysis even if collinearity is NOT an issue. You may find, for example, that a smaller model may have a better/equivalent AIC or Adjusted $R^2$ than the full model.

$\endgroup$
3
  • $\begingroup$ The apparent improvement in a model by doing variable selection is a complete mirage. It results by not respecting model uncertainty, e.g., by using the apparent degrees of freedom in the AIC calculation and not the effective degrees of freedom. $\endgroup$ Nov 17, 2021 at 16:48
  • $\begingroup$ Is it not situational? For simple linear models, I think it is worth doing some variable selection. I think you're absolutely right in more complex cases. $\endgroup$
    – decaf
    Nov 18, 2021 at 18:01
  • $\begingroup$ In the best case for variable selection, variable selection is still very problematic. Why do it? $\endgroup$ Nov 18, 2021 at 21:49
0
$\begingroup$

Something fancier than a univariable or multiple regression model is needed when there is a very clear, very complicated non-linear relationship between the endpoint and covariates that cannot be addressed using a routine link function or transformed covariate. If your paneled scatter plots show clouds of data points with suggestive trends then anything fancier could be considered overfitting.

Based on your labeling of what is the dependent and independent variables, your model will tell you if any DV's (high or low) are more common in words with certain properties. Your description had this reversed, suggesting reversal of dependent and independent variables.

Yes, with a large enough sample some of your predictors are bound to be statistically significant, and that is a good thing. In fact, it is a great thing. Don't be disappointed with small p-values. However, just because you have the power to detect an effect does not imply it is a meaningful effect to discuss and report.

As decaf and Eoin have suggested, you can use LASSO and FDR methods among others to weed out independent variables. My preference is to use the p-value as a continuous index for the weight of the evidence by running univariable models and sorting the predictors by ascending p-value, as well as by the magnitude of the estimated effect size. Those predictors with the smallest p-values do the best job explaining your endpoint for your particular data set and stand the best chance of re-demonstrating their effects in a repeated experiment. If this "short list" is still a bit too long you can choose to highlight only the top, say, $k$ independent variables on the list. If we think in terms of Neyman-Pearson error rates instead of evidential p-values, we can talk in terms of per-comparison error rates so that no adjustments to the p-values are necessary, acknowledging that family-wise error rates may very well be higher. The key is to interpret the results as tentative evidence, not irrefutable proof. Those independent variables with the largest effect sizes are the most interesting to discuss and report, even if the p-value does not reach a conventional cutoff like 0.05.

The independent variables that were promising in univariable models can be explored together in a multivariable model to see if the conditional effects persist in smaller subgroups. I suggest creating paneled and grouped plots to explore possible interaction effects. However, this can become very nebulous very quickly. For that reason I prefer to keep emphasis on the univariable analyses that describe the higher-level conditional effect for each independent variable. This may seem elementary, but it may be naive to slice and dice the data by a dozen or more independent variables simultaneously and discuss a miniscule effect in an obscure subgroup especially when the results are tentative evidence and not irrefutable proof. It may be worthwhile to define a subpopulation and fit a simpler model to the data available on this subpopulation, rather than trying to interpret a multivariable regression model. Such a stratified model may provide a better fit to the data and the reduced sample size will produce more conservative inference.

No matter what modeling technique you choose the only true validation is to repeat the experiment and see if the same model does well at explaining the dependent variable. If the experiment can be replicated many times then what was tentative evidence, taken together, becomes irrefutable evidence.

$\endgroup$
7
  • $\begingroup$ I wonder if there is much value in running univariable models [and sorting the predictors by ascending p-value], since I doubt the following holds: Those predictors with the smallest p-values do the best job explaining your endpoint for your particular data set and stand the best chance of re-demonstrating their effects in a repeated experiment. $\endgroup$ Nov 17, 2021 at 9:57
  • $\begingroup$ Hi Richard, I am sure you understand my point quite well. I am using the phrase, "explaining your endpoint," to mean fitting the data, much like a model with a large $R^2$ or small AIC fits the data well. There of course is no guarantee of reproducibility, but if the sample is representative of the population then such a model should also fit well in repeated samples. If you have a particular suggestion for improving my answer please let me know. $\endgroup$ Nov 17, 2021 at 12:02
  • $\begingroup$ My concern regards using $k$ simple regressions instead of a multiple regression with $k$ explanatory variables and expecting the former to be superior. $\endgroup$ Nov 17, 2021 at 12:26
  • $\begingroup$ Ah, I see your point. I agree that $k$ univariable regression models are not superior than a multiple regression for explaining the conditional mean across multiple subgroups. The univariable and multivariable approaches are indeed providing two different answers to two different questions. I am suggesting that we shouldn't fit complicated multivariable models out of a need to feel fancy or sophisticated in our modeling technique, and that higher-level relationships are often what we are most interested in. $\endgroup$ Nov 17, 2021 at 12:36
  • $\begingroup$ No this discussion is getting off track. Looking at p-values is a recipe for disaster. And univariable screening is a train wreck. $\endgroup$ Nov 17, 2021 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.