Coefficients returned from regression model don't seem right

Question

I'm using sklearn's linearRegression model. After regression is complete, I get back a set of features and a set of coefficients. Referring to this post, I found how to map each feature to its corresponding coefficient. If I understand correctly, these coefficients are the coefficient of variables in the equation that represents the independent variable.

So, for instance:

If I am modeling the selling price of a toy based on time since launch (how many days ago was this toy released in the market) and weight of toy

and

i get -0.3 for time_since_launch and +200 for weight,

then weight is the dominating factor.

If this understanding is correct, I am confused about the results that my model is giving me.

My methodology:

I am performing stepwise linear regression (I know it has flaws, but I expect it to still give me reasonable results). Each iteration of the stepwise process iterates through each of the 150 features, performs regression with the {current set of features+ current loop feature} and then includes the feature that gave the lowest loss.

Now, the stepwise iteration is spitting out features in an expected order, i.e., features that I expected to have high impact were being added to the model first. However, when I mapped the coefficients and feature names, something seemed very off. Some of the features that were added to the model early on, had very low coefficients (and some features that were added later, had high coefficients).

Can anyone spot the problem?

Answer 1

Variable importance is not straightforward in linear regression since variables might be correlated.

However, its a good practice to do the following when preparing your data for training models:

1. Ensure all variables representing the same quantities are in the same units (all distance in meters, weights in grams, or as per the scale suitable to the problem)
2. Scale all variables to zero mean and unit standard deviation

Determining the importance of variables in linear regression

A popular approach to determine variable importance for linear regression models is to decompose the $R^2$ into contributions attributed to each variable.

Refer to the document describing the PMD method (Feldman, 2005). Another popular approach is averaging over orderings (LMG, 1980). The LMG works like this:

• Find the semi-partial correlation of each predictor in the model, e.g. for variable a we have: $SS_a/SS_{total}$. It implies how much would $R^2$ increase if variable $a$ were added to the model.
• Calculate this value for each variable for each order in which the variable gets introduced into the model, i.e. {$a,b,c$} ; {$b,a,c$} ; {$b,c,a$}
• Find the average of the semi-partial correlations for each of these orders. This is the average over orderings.

The R package relaimpo implements 6 different metrics for assessing relative importance of variables in the linear regression model, including averaging over orderings of regressors and pmvd. relaimpo also gives bootstrap confidence intervals.

References:

• Relative importance of Linear Regressors in R
• Relative Importance and Value, Barry Feldman (PMD method)