7
$\begingroup$

My goal is to understand which variables out of 10 actually explain my dependent variable for a voter/election research.
Now I have put all 10 variables into a multiple regression and have filtered out all that have p value over 0.1. Now I have a model with 3 variables, all p values at 0.014 or far below and r=0.8. It seems all fine.

Can't I just use this model? Or do I absolutely have to use lasso? Because I have tried to bring lasso to work for hours now. I can't do it (only experienced in excel). It's really frustrating all the errors R throws at me. If I have to use lasso, how can I do this the easiest? Is there any way to do LASSO or something similar with excel?

$\endgroup$
8
  • 4
    $\begingroup$ "do I absolutely have to use lasso?" no, and I'm sure we're all wondering where on earth you got this idea :) May be worth checking out this Wikipedia page which lists some variable selection methods (only some of which are lasso ;) en.wikipedia.org/wiki/Feature_selection) $\endgroup$ Dec 25, 2022 at 3:26
  • $\begingroup$ What is your sample size? $\endgroup$ Dec 25, 2022 at 10:00
  • $\begingroup$ @christian My sample size (number of observations) is 110 and as more data is coming in, it will be 250. $\endgroup$ Dec 25, 2022 at 11:05
  • 1
    $\begingroup$ In comments the OP clarifies that they are interested in "understand(ing) real world relationships. (...) I think they are layered effects. Like x1 and x2 directly influence Y, but x1 itself is influenced by x4 and x7." The language of "direct influence" suggests that you really want to do is causal inference. If that's the case, feature selection is basically irrelevant and in fact you cannot build the causal graph from the data. Instead, you could use the prior domain knowledge to construct the causal graph and start from there. $\endgroup$
    – dipetkov
    Dec 25, 2022 at 14:53
  • 1
    $\begingroup$ There are many CV threads that might be of interest. For example: Causal Inference After Feature Selection and A more definitive discussion of variable selection. $\endgroup$
    – dipetkov
    Dec 25, 2022 at 14:57

4 Answers 4

9
$\begingroup$

Variable selection is not compulsory. The idea that you have to throw away variables is wrong. Actually, unless there are strong reasons to throw away variables, don't do it! Use the full model and use the p-values to guide interpretation rather than throwing away information. An insignificant p-value doesn't mean that a variable should be removed, it only means that the data don't give you clear evidence that the coefficient is nonzero.

The Lasso mitigates some of the issues of doing variable selection by p-values, but as long as you don't feel the need to remove information, there's no need to do it, neither by Lasso.

Good reasons for selecting variables are:

  1. The number of observations is critically low for the number of variables you have.

  2. There are strong dependencies between your variables ($X^TX$ is close to singular) and certain information is represented by several variables

  3. The model is used to predict future observations and you are happy to get rid of some variables because it may be costly to observe them in the future.

  4. Using cross-validation and the like you find that a model with fewer variables predicts better (if this happens, mostly one of 1 and 2 is also in place, but there are some further situations in which the original set of variables contains a lot of noise).

Variable selection is not required for interpretation! Note that even if there is no evidence (high p-value) that a certain variable has a nonzero coefficient, this variable may still improve the predictive power of the model. Removing the variable will set its coefficient to zero - don't forget that the estimator for it's coefficient in the full model is the "best guess" of the coefficient value that you have, significant or not, and therefore also better than zero in the sense of least squares.

By the way, regarding Lasso vs. variable selection by p-values (backward/forward/stepwise selection - never throw away all variables with large p-values in one go anyway): There are well known issues with variable selection by p-values and Lasso is often better, but not always. I've seen a good number of examples in which a model selected by backward or forward selection did better predicting on independent data than the Lasso. If you have enough observations to properly assess prediction quality using cross-validation and the like, you can compare different approaches and pick the best rather than always using Lasso.

$\endgroup$
2
  • 1
    $\begingroup$ Thanks for your thorough answer and Merry Christmas! I think I may have phrased the question not precicely enough. Yes my goal is to understand real world relationships. More precisely: I think the independent variables are not just correlated by chance. I think they are layered effects. Like x1 and x2 directly influence Y, but x1 itself is influenced by x4 and x7. I think that's the case because all independend variables are correlated and because of prior theoretic research. That's why I did the stepwise. Do you know what to Google to get help with that problem? $\endgroup$ Dec 25, 2022 at 11:57
  • 1
    $\begingroup$ @CrossVal_Asker I don't think variable selection will help with your problem. You may rather be interested in causal inference using graphical models. $\endgroup$ Dec 25, 2022 at 16:37
6
$\begingroup$

My goal is to understand which variables out of 10 actually explain my dependent variable for a voter/election research.

This is not variable selection!

With variable selection the goal is to reduce the size of a model

  • Such that it is easier to work with, like less computation intensive or requires less future efforts in sampling.
  • Such that it is less sensitive to noise. See 'bias variance' tradeoff.

Your goal is to understand a relationship. This can be done by making a plot and observe a straight line (or in the case of more variables the line is a plane and you look at statistics in a table instead of a figure because plotting 10-D is not easy in our 3-D physical world).

The role of the p-values is to filter the effects from the noise (like you would do in the 2-D, plot to tell if there is a straight line relationship, or just noise). The p-values indicate statistical significance. That means that you have observed an effect that is statistically measurable with your experiment. The other effects, with high p-value, might be present but are overshadowed by noise.

So in your case it seems fine to go with p-values. However, be aware of the pitfalls with p-values.

$\endgroup$
1
  • 1
    $\begingroup$ Many Thanks for your thorough answer :)! So "variable selection" and "understanding real world relationships in the model are two different fields, correct? If yes, we are coming closer to the heart of the problem: Yes my goal is to understand relationships. More precisely: I think the independent variables are not just correlated by chance. I think they are layered effects. Like x1 and x2 directly influence Y, but x1 itself is influenced by x4 and x7. That's why I did the stepwise. Do you know what to Google to get help with that problem? $\endgroup$ Dec 25, 2022 at 11:55
3
$\begingroup$

If its only the Feature Importance Aspect you are looking for , you can use RFE Method (Recursive Feature Elimination) as well.

But Lasso Regression takes care of Model not reaching Overfit or Underfit Situations as well as Feature Importance, by reducing all the Useless Features to Zero.

You can use Lasso Regression using GridSearchCV method, and then plot your Mean Train and Mean Test Errors accordingly, if you have different values of Regularization Parameters available. And for the best values of Errors, you can consider the value for the Regularization Parameter and fit your model with Train Data.

$\endgroup$
0
$\begingroup$

For any others that might be interested in the answer: I managed to do the LASSO with an excel plugin into excel 365. It did show significantly different results in comparison to stepwise. Also, I found out that I actually want to build a causal model not a variable selection. (different terminology). So yes LASSO seems to be worth the work.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.