4
$\begingroup$

I am running a lasso regression function. I have about 45 features and I am predicting 1 dependent variable. After running lasso regression I get the coefficient values of the features.

If I look at the magnitude of the coefficients do they tell me how important the respective feature was for prediction? for example a feature with a coefficient=100 has more predictive power/importance than one with a value if 20 or 0.

$\endgroup$
5
$\begingroup$

You cannot compare the values of coefficients in this way. Suppose that your response $Y$ is measured in meters, and you have two features $X_1$ and $X_2$ which are measured in seconds and hours respectively. Then your coefficients: $\beta_1$ has units meters/second and $\beta_2$ has units meters/hour - these are not comparable directly. Even worse is if $X_1$ is measured in seconds but $X_2$ is something totally unrelated, say ohms, coulombs, newtons or lumens.

Now, when doing lasso regression, it is standard practice to standardize the columns in the design matrix, which essentially makes all the predictors dimensionless (though when the coefficients are reported back to the user, they are usually stated on the original scale). You still cannot compare the magnitudes in any reasonable way. A simple way to see this is to consider the following situation:

$$ \begin{align*} Y = X_1 + X_2 + \epsilon \\ corr(X_1, X_2) = 1 \end{align*} $$

Any of the following regression models is correct:

$$ \begin{align*} E(Y \mid X_1, X_2) &= X_1 + X_2 \\ E(Y \mid X_1, X_2) &= 2 X_1 \\ E(Y \mid X_1, X_2) &= 2 X_2 \\ E(Y \mid X_1, X_2) &= .5 X_1 + 1.5 X_2 \end{align*} $$

and so on. Of course, situations found "in nature" are never this clear cut, but this illustrates the essential difficulties in your proposal.

$\endgroup$
  • 1
    $\begingroup$ What do you think of keeping the top, say, 30% of coef, ranked by magnitude? Are such practices common in using lasso? $\endgroup$ – mac Oct 28 '16 at 16:17
  • 2
    $\begingroup$ If you are using LASSO for feature selection, you usually employ cross-validation for selection of a lambda value based on your metric of interest (e.g., accuracy, logloss, etc.). Then you keep all covariates not set to zero based on the selected lambda value. You don't need to utilize an arbitrary percentage # feature to keep, since some of those may not be informative. $\endgroup$ – hlsmith Jul 5 '18 at 17:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.