# For linear classifiers, do larger coefficients imply more important features?

I'm a software engineer working on machine learning. From my understanding, linear regression (such as OLS) and linear classification (such as logistic regression and SVM) make a prediction based on an inner product between trained coefficients $\vec{w}$ and feature variables $\vec{x}$:

$$\hat{y} = f(\vec{w} \cdot \vec{x}) = f(\sum_{i} w_i x_i)$$

My question is: After the model has been trained (that is, after the coefficients $w_i$ have be computed), is it the case that the coefficients will be larger for feature variables that are more important for the model to predict more accurately?

In other words, I am asking whether the relative magnitudes of the coefficients can be used for feature selection by just ordering the variables by coefficient value and then selecting the features with the highest coefficients? If this approach is valid, then why is it not mentioned for feature selection (along with wrapper and filter methods, etc.).

The reason I ask this is because I came across a discussion on L1 vs. L2 regularization. There is a blurb that says:

Built-in feature selection is frequently mentioned as a useful property of the L1-norm, which the L2-norm does not. This is actually a result of the L1-norm, which tends to produces sparse coefficients (explained below). Suppose the model have 100 coefficients but only 10 of them have non-zero coefficients, this is effectively saying that "the other 90 predictors are useless in predicting the target values".

Reading between the lines, I would guess that if a coefficient is close to 0, then the feature variable with that coefficient must have little predictive power.

EDIT: I am also applying z-scaling to my numeric variables.

• Note that the code underlying LASSO (L1-norm) and ridge regression (L2-norm) analyses should pre-scale the predictor variables prior to analysis, even if the code then transforms the coefficients back into the original variable scales. Those who use code that doesn't pre-scale end up with the problems noted in the answer from @josliber whether they are doing OLS, LASSO, or ridge. – EdM Mar 21 '16 at 18:30
• I think something worth mentioning is, when you reflect on what are trying to express by the phrase "then the feature variable with that coefficient must have little predictive power", can you precisely lay out what that really means? I've found though experience that the concept of "predictive power" of an individual variable in a multivariate model has no generally agreed upon conceptual foundation. – Matthew Drury Mar 21 '16 at 18:55
• I think the error in that kind of thinking is that you are probably not confined to producing a one variable model. If you are, and you want to provide a model with the best accuracy, they sure, that is a reasonable thing to do. If you are not, i.e. if you are going to produce a multivariate model, then, as @EdM answers, the concept of variable importance is very, very slippery, and lacks a firm conceptual foundation. It is not at all obvious that predictive power in a univariate model should seen as relevant in a multivariate setting. – Matthew Drury Mar 21 '16 at 20:36
• @MatthewDrury: I'm not sure why you are making a big deal out of multi-features. There is a whole field of "feature selection" (e.g. wrapper methods) that exists; are you suggesting that this field lacks a firm conceptual foundation? – stackoverflowuser2010 Mar 22 '16 at 17:54
• @stackoverflowuser2010 Yah, I'm probably an outlier in my opinion here, but that would be a somewhat accurate description of my perspective. – Matthew Drury Mar 22 '16 at 21:21

Not at all. The magnitude of the coefficients depends directly on the scales selected for the variables, which is a somewhat arbitrary modeling decision.

To see this, consider a linear regression model predicting the petal width of an iris (in centimeters) given its petal length (in centimeters):

summary(lm(Petal.Width~Petal.Length, data=iris))
# Call:
# lm(formula = Petal.Width ~ Petal.Length, data = iris)
#
# Residuals:
#      Min       1Q   Median       3Q      Max
# -0.56515 -0.12358 -0.01898  0.13288  0.64272
#
# Coefficients:
#               Estimate Std. Error t value Pr(>|t|)
# (Intercept)  -0.363076   0.039762  -9.131  4.7e-16 ***
# Petal.Length  0.415755   0.009582  43.387  < 2e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#
# Residual standard error: 0.2065 on 148 degrees of freedom
# Multiple R-squared:  0.9271,  Adjusted R-squared:  0.9266
# F-statistic:  1882 on 1 and 148 DF,  p-value: < 2.2e-16


Our model achieves an adjusted R^2 value of 0.9266 and assigns coefficient value 0.415755 to the Petal.Length variable.

However, the choice to define Petal.Length in centimeters was quite arbitrary, and we could have instead defined the variable in meters:

iris$Petal.Length.Meters <- iris$Petal.Length / 100
summary(lm(Petal.Width~Petal.Length.Meters, data=iris))
# Call:
# lm(formula = Petal.Width ~ Petal.Length.Meters, data = iris)
#
# Residuals:
#      Min       1Q   Median       3Q      Max
# -0.56515 -0.12358 -0.01898  0.13288  0.64272
#
# Coefficients:
#                     Estimate Std. Error t value Pr(>|t|)
# (Intercept)         -0.36308    0.03976  -9.131  4.7e-16 ***
# Petal.Length.Meters 41.57554    0.95824  43.387  < 2e-16 ***
# ---
# Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
#
# Residual standard error: 0.2065 on 148 degrees of freedom
# Multiple R-squared:  0.9271,  Adjusted R-squared:  0.9266
# F-statistic:  1882 on 1 and 148 DF,  p-value: < 2.2e-16


Of course, this doesn't really affect the fitted model in any way -- we simply assigned a 100x larger coefficient to Petal.Length.Meters (41.57554) than we did to Petal.Length (0.415755). All other properties of the model (adjusted R^2, t-statistics, p-values, etc.) are identical.

Generally when fitting regularized linear models one will first normalize variables (for instance, to have mean 0 and unit variance) to avoid favoring some variables over others based on the selected scales.

## Assuming Normalized Data

Even if you had normalized all variables, variables with higher coefficients may still not be as useful in predictions because the independent variables are rarely set (have low variance). As an example, consider a dataset with dependent variable Z and independent variables X and Y taking binary values

set.seed(144)
dat <- data.frame(X=rep(c(0, 1), each=50000),
Y=rep(c(0, 1), c(1000, 99000)))