2
$\begingroup$

I have an experiment data (total of 96) with 10 predictor and 2 response variables. I want to build a traditional multiple linear regression model to them in R. My aim is to build clearly interpretable (coefficients) and explanatory models for a journal article. Not all variables being significant, I have been trying to reduce the number of variables by checking their t and f table p values as well as the VIF scores to reduce multicollinearity among them. However, rather than this manual selection, I want to use a straightforward and commonly accepted automated feature selection.

I have been doing stepwise regression and I stopped after reading all the statistical doubts about it in the forum. I have read many people suggest Lasso models or some more machine learning related sophisticated approach for this. But, I want to keep this process as simple as possible such that the people in my area (engineering) can easily understand it.

Does anyone have any other suggestion rather than stepwise regression for this automated process? Note: I have realized some interaction variables play important role in my dataset, so I want to include them as well into this trial (even though the multicollinearity coming with it).

$\endgroup$
3
  • $\begingroup$ Why select features at all? Linear regression is already interpretable, what do you gain by selecting features via a data first approach and not using your domain expertise? $\endgroup$ Oct 31, 2023 at 15:46
  • $\begingroup$ Because I want the simplest model (principle of parsimony). Some variables are not significantly affecting the model and there are multicollinearity within the variables such that I want to carefully remove some of them by not only my domain expertise but also an accepted automation algorithm. $\endgroup$
    – egeku
    Oct 31, 2023 at 16:27
  • $\begingroup$ There is no "accepted automation algorithm" because data are just numbers without context (that is to say, your domain knowledge). Coefficients failing to reject the null and multicollinearity are not as problematic as one thinks. What is your experiment on? What is the goal of this experiment? Let's start there and maybe we can find a suitable solution. $\endgroup$ Oct 31, 2023 at 16:52

1 Answer 1

1
$\begingroup$

See Frank Harrell's Regression Modeling Strategies for extensive discussion about how to cut down on the number of predictors responsibly. Any method that uses the outcomes to select or combine predictors will lead to the same problems as stepwise regression. You certainly can, however, work with the predictors themselves to combine your domain expertise with methods that reduce their effective numbers.

If you have collinear continuous predictors, I suspect that your audience of engineers would be able to understand and accept an approach based on principal components (PCs), combining several predictors that are correlated and only keeping the first 1 or 2 PCs. You can choose some illustrative name for those PCs based on your domain knowledge. Section 4.7 goes into further "data reduction" approaches, based for example on redundancy analysis or predictor clustering. Chapter 8 illustrates the principles in a case study.

Those might not be the "automated" approaches that you seem to want, but these approaches avoid the major pitfall of automated approaches that use the outcomes to pick the predictors.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.