14
$\begingroup$

I'm a software engineer working on machine learning. From my understanding, linear regression (such as OLS) and linear classification (such as logistic regression and SVM) make a prediction based on an inner product between trained coefficients $\vec{w}$ and feature variables $\vec{x}$:

$$ \hat{y} = f(\vec{w} \cdot \vec{x}) = f(\sum_{i} w_i x_i) $$

My question is: After the model has been trained (that is, after the coefficients $w_i$ have be computed), is it the case that the coefficients will be larger for feature variables that are more important for the model to predict more accurately?

In other words, I am asking whether the relative magnitudes of the coefficients can be used for feature selection by just ordering the variables by coefficient value and then selecting the features with the highest coefficients? If this approach is valid, then why is it not mentioned for feature selection (along with wrapper and filter methods, etc.).

The reason I ask this is because I came across a discussion on L1 vs. L2 regularization. There is a blurb that says:

Built-in feature selection is frequently mentioned as a useful property of the L1-norm, which the L2-norm does not. This is actually a result of the L1-norm, which tends to produces sparse coefficients (explained below). Suppose the model have 100 coefficients but only 10 of them have non-zero coefficients, this is effectively saying that "the other 90 predictors are useless in predicting the target values".

Reading between the lines, I would guess that if a coefficient is close to 0, then the feature variable with that coefficient must have little predictive power.

EDIT: I am also applying z-scaling to my numeric variables.

$\endgroup$
  • 1
    $\begingroup$ Note that the code underlying LASSO (L1-norm) and ridge regression (L2-norm) analyses should pre-scale the predictor variables prior to analysis, even if the code then transforms the coefficients back into the original variable scales. Those who use code that doesn't pre-scale end up with the problems noted in the answer from @josliber whether they are doing OLS, LASSO, or ridge. $\endgroup$ – EdM Mar 21 '16 at 18:30
  • 3
    $\begingroup$ I think something worth mentioning is, when you reflect on what are trying to express by the phrase "then the feature variable with that coefficient must have little predictive power", can you precisely lay out what that really means? I've found though experience that the concept of "predictive power" of an individual variable in a multivariate model has no generally agreed upon conceptual foundation. $\endgroup$ – Matthew Drury Mar 21 '16 at 18:55
  • 4
    $\begingroup$ I think the error in that kind of thinking is that you are probably not confined to producing a one variable model. If you are, and you want to provide a model with the best accuracy, they sure, that is a reasonable thing to do. If you are not, i.e. if you are going to produce a multivariate model, then, as @EdM answers, the concept of variable importance is very, very slippery, and lacks a firm conceptual foundation. It is not at all obvious that predictive power in a univariate model should seen as relevant in a multivariate setting. $\endgroup$ – Matthew Drury Mar 21 '16 at 20:36
  • 1
    $\begingroup$ @MatthewDrury: I'm not sure why you are making a big deal out of multi-features. There is a whole field of "feature selection" (e.g. wrapper methods) that exists; are you suggesting that this field lacks a firm conceptual foundation? $\endgroup$ – stackoverflowuser2010 Mar 22 '16 at 17:54
  • 1
    $\begingroup$ @stackoverflowuser2010 Yah, I'm probably an outlier in my opinion here, but that would be a somewhat accurate description of my perspective. $\endgroup$ – Matthew Drury Mar 22 '16 at 21:21
24
+25
$\begingroup$

Not at all. The magnitude of the coefficients depends directly on the scales selected for the variables, which is a somewhat arbitrary modeling decision.

To see this, consider a linear regression model predicting the petal width of an iris (in centimeters) given its petal length (in centimeters):

summary(lm(Petal.Width~Petal.Length, data=iris))
# Call:
# lm(formula = Petal.Width ~ Petal.Length, data = iris)
# 
# Residuals:
#      Min       1Q   Median       3Q      Max 
# -0.56515 -0.12358 -0.01898  0.13288  0.64272 
# 
# Coefficients:
#               Estimate Std. Error t value Pr(>|t|)    
# (Intercept)  -0.363076   0.039762  -9.131  4.7e-16 ***
# Petal.Length  0.415755   0.009582  43.387  < 2e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
# 
# Residual standard error: 0.2065 on 148 degrees of freedom
# Multiple R-squared:  0.9271,  Adjusted R-squared:  0.9266 
# F-statistic:  1882 on 1 and 148 DF,  p-value: < 2.2e-16

Our model achieves an adjusted R^2 value of 0.9266 and assigns coefficient value 0.415755 to the Petal.Length variable.

However, the choice to define Petal.Length in centimeters was quite arbitrary, and we could have instead defined the variable in meters:

iris$Petal.Length.Meters <- iris$Petal.Length / 100
summary(lm(Petal.Width~Petal.Length.Meters, data=iris))
# Call:
# lm(formula = Petal.Width ~ Petal.Length.Meters, data = iris)
# 
# Residuals:
#      Min       1Q   Median       3Q      Max 
# -0.56515 -0.12358 -0.01898  0.13288  0.64272 
# 
# Coefficients:
#                     Estimate Std. Error t value Pr(>|t|)    
# (Intercept)         -0.36308    0.03976  -9.131  4.7e-16 ***
# Petal.Length.Meters 41.57554    0.95824  43.387  < 2e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
# 
# Residual standard error: 0.2065 on 148 degrees of freedom
# Multiple R-squared:  0.9271,  Adjusted R-squared:  0.9266 
# F-statistic:  1882 on 1 and 148 DF,  p-value: < 2.2e-16

Of course, this doesn't really affect the fitted model in any way -- we simply assigned a 100x larger coefficient to Petal.Length.Meters (41.57554) than we did to Petal.Length (0.415755). All other properties of the model (adjusted R^2, t-statistics, p-values, etc.) are identical.

Generally when fitting regularized linear models one will first normalize variables (for instance, to have mean 0 and unit variance) to avoid favoring some variables over others based on the selected scales.

Assuming Normalized Data

Even if you had normalized all variables, variables with higher coefficients may still not be as useful in predictions because the independent variables are rarely set (have low variance). As an example, consider a dataset with dependent variable Z and independent variables X and Y taking binary values

set.seed(144)
dat <- data.frame(X=rep(c(0, 1), each=50000),
                  Y=rep(c(0, 1), c(1000, 99000)))
dat$Z <- dat$X + 2*dat$Y + rnorm(100000)

By construction, the coefficient for Y is roughly twice as large as the coefficient for X when both are used to predict Z via linear regression:

summary(lm(Z~X+Y, data=dat))
# Call:
# lm(formula = Z ~ X + Y, data = dat)
# 
# Residuals:
#     Min      1Q  Median      3Q     Max 
# -4.4991 -0.6749 -0.0056  0.6723  4.7342 
# 
# Coefficients:
#              Estimate Std. Error t value Pr(>|t|)    
# (Intercept) -0.094793   0.031598   -3.00   0.0027 ** 
# X            0.999435   0.006352  157.35   <2e-16 ***
# Y            2.099410   0.031919   65.77   <2e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
# 
# Residual standard error: 0.9992 on 99997 degrees of freedom
# Multiple R-squared:  0.2394,  Adjusted R-squared:  0.2394 
# F-statistic: 1.574e+04 on 2 and 99997 DF,  p-value: < 2.2e-16

Still, X explains more of the variance in Z than Y (the linear regression model predicting Z with X has R^2 value 0.2065, while the linear regression model predicting Z with Y has R^2 value 0.0511):

summary(lm(Z~X, data=dat))
# Call:
# lm(formula = Z ~ X, data = dat)
# 
# Residuals:
#     Min      1Q  Median      3Q     Max 
# -5.2587 -0.6759  0.0038  0.6842  4.7342 
# 
# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept) 1.962629   0.004564   430.0   <2e-16 ***
# X           1.041424   0.006455   161.3   <2e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
# 
# Residual standard error: 1.021 on 99998 degrees of freedom
# Multiple R-squared:  0.2065,  Adjusted R-squared:  0.2065 
# F-statistic: 2.603e+04 on 1 and 99998 DF,  p-value: < 2.2e-16

versus:

summary(lm(Z~Y, data=dat))
# Call:
# lm(formula = Z ~ Y, data = dat)
# 
# Residuals:
#     Min      1Q  Median      3Q     Max 
# -5.0038 -0.7638 -0.0007  0.7610  5.2288 
# 
# Coefficients:
#             Estimate Std. Error t value Pr(>|t|)    
# (Intercept) -0.09479    0.03529  -2.686  0.00724 ** 
# Y            2.60418    0.03547  73.416  < 2e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
# 
# Residual standard error: 1.116 on 99998 degrees of freedom
# Multiple R-squared:  0.05114, Adjusted R-squared:  0.05113 
# F-statistic:  5390 on 1 and 99998 DF,  p-value: < 2.2e-16

The Case of Multi-Collinearity

A third case where large coefficient values may be deceiving would be in the case of significant multi-collinearity between variables. As an example, consider a dataset where X and Y are highly correlated but W is not highly correlated to the other two; we are trying to predict Z:

set.seed(144)
dat <- data.frame(W=rnorm(100000),
                  X=rnorm(100000))
dat$Y <- dat$X + rnorm(100000, 0, 0.001)
dat$Z <- 2*dat$W+10*dat$X-11*dat$Y + rnorm(100000)
cor(dat)
#              W             X             Y          Z
# W 1.000000e+00  5.191809e-05  5.200434e-05  0.8161636
# X 5.191809e-05  1.000000e+00  9.999995e-01 -0.4079183
# Y 5.200434e-05  9.999995e-01  1.000000e+00 -0.4079246
# Z 8.161636e-01 -4.079183e-01 -4.079246e-01  1.0000000

These variables pretty much have the same mean (0) and variance (~1), and linear regression assigns much higher coefficient values (in absolute value) to X (roughly 15) and Y (roughly -16) than it does to W (roughly 2):

summary(lm(Z~W+X+Y, data=dat))
# Call:
# lm(formula = Z ~ W + X + Y, data = dat)
# 
# Residuals:
#     Min      1Q  Median      3Q     Max 
# -4.1886 -0.6760  0.0026  0.6679  4.2232 
# 
# Coefficients:
#               Estimate Std. Error t value Pr(>|t|)    
# (Intercept)  1.831e-04  3.170e-03   0.058    0.954    
# W            2.001e+00  3.172e-03 630.811  < 2e-16 ***
# X            1.509e+01  3.177e+00   4.748 2.05e-06 ***
# Y           -1.609e+01  3.177e+00  -5.063 4.13e-07 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
# 
# Residual standard error: 1.002 on 99996 degrees of freedom
# Multiple R-squared:  0.8326,  Adjusted R-squared:  0.8326 
# F-statistic: 1.658e+05 on 3 and 99996 DF,  p-value: < 2.2e-16

Still, among the three variables in the model W is the most important: If you remove W from the full model, the R^2 drops from 0.833 to 0.166, while if you drop X or Y the R^2 is virtually unchanged.

$\endgroup$
  • 1
    $\begingroup$ (+1) I think this is a good answer, and this point must be made. On the other hand, I do believe there is a lot to say even when the variables have been standardized (and are hence, unit-less), so there's room for a few more answers. $\endgroup$ – Matthew Drury Mar 21 '16 at 18:31
  • $\begingroup$ Thanks for the answer. While your writing is long, it's not very thorough because you're drawing conclusions from small synthetic data. Also, R^2 is specific to linear regression. I believe the more appropriate error metric is RMSE, or accuracy / F1 for classification problems. $\endgroup$ – stackoverflowuser2010 Mar 29 '16 at 2:36
  • 1
    $\begingroup$ BUT if you scale the data in the last example, you have that the only significant variable is W $\endgroup$ – marcodena Aug 16 '16 at 17:40
11
$\begingroup$

"Feature importance" is a very slippery concept even when all predictors have been adjusted to a common scale (which in itself is a non-trivial problem in many practical applications involving categorical variables or skewed distributions). So if you avoid the scaling problems indicated in the answer by @josliber or the low-predictor-variance issue raised by @dsaxton, you still have additional problems.

For example, a more useful measure of feature importance may be the ratio of its coefficient to the estimated error of its coefficient. A high coefficient with a large estimated error would not necessarily be helpful in predictions. So coefficient magnitude alone, even in the pre-scaled situation, is not a good guide to "importance."

Nevertheless, a predictor may be important even if its coefficient's ratio of magnitude to error is low (i.e., it is not "statistically significant"). Linear models provide the ability to take multiple predictor variables into account simultaneously, so including a "non-significant" predictor in a model can improve the overall performance provided by the combined collection of predictors.

Furthermore, attempts to select "important" predictor variables tend to be highly dependent on the particular data sample and often do not extend well to further samples, particularly if the variables are correlated. You can see this for yourself by repeating feature selection on multiple bootstrap samples of the same data set. Frank Harrell, in this answer shows how to use his rms package in R for ranking feature importance, and notes in this answer how to use the bootstrap to get confidence intervals for the ranks. Bootstrapping can serve as a caution to those who put too much importance in "feature importance."

This question from nearly 3 years ago, pointed out by @amoeba, also goes into extensive detail about difficulties with feature importance in multiple regression models.

$\endgroup$
  • $\begingroup$ Ratio of estimated error. Is this known as "standardized coefficient"? $\endgroup$ – SmallChess Nov 24 '16 at 6:51
  • $\begingroup$ @StudentT a "standardized coefficient" is a regression coefficient when the independent and dependent variables have all been scaled to have unit variance. That includes no information about the estimated error in the coefficient. The ratio I describe is the square root of the Wald statistic used by Harrell as a variable-importance measure in the first of my links. $\endgroup$ – EdM Nov 24 '16 at 12:51
6
$\begingroup$

Just to add to the previous answer, the coefficient itself also fails to capture how much variability a predictor exhibits, which has a large effect on how useful it is in making predictions. Consider the simple model

$$ \text{E}(Y_i) = \alpha + \beta X_i $$

where $X_i$ is a Bernoulli$(p)$ random variable. By taking $p \to 0$ we can send the usefulness of this predictor to zero as well, but the coefficient will always be $\beta$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.