I'm using sklearn's linearRegression model. After regression is complete, I get back a set of features and a set of coefficients. Referring to this post, I found how to map each feature to its corresponding coefficient. If I understand correctly, these coefficients are the coefficient of variables in the equation that represents the independent variable.

So, for instance:

If I am modeling the selling price of a toy based on time since launch (how many days ago was this toy released in the market) and weight of toy


i get -0.3 for time_since_launch and +200 for weight,

then weight is the dominating factor.

If this understanding is correct, I am confused about the results that my model is giving me.

My methodology:

I am performing stepwise linear regression (I know it has flaws, but I expect it to still give me reasonable results). Each iteration of the stepwise process iterates through each of the 150 features, performs regression with the {current set of features+ current loop feature} and then includes the feature that gave the lowest loss.

Now, the stepwise iteration is spitting out features in an expected order, i.e., features that I expected to have high impact were being added to the model first. However, when I mapped the coefficients and feature names, something seemed very off. Some of the features that were added to the model early on, had very low coefficients (and some features that were added later, had high coefficients).

Can anyone spot the problem?

  • 2
    $\begingroup$ In the toy example you give, the regression should return very different coefficient values if time is in seconds, days, or years - and if weight is in grams, kilograms, or metric tons. If all data is re-scaled from zero to one, the weights are more comparable. $\endgroup$ – James Phillips Nov 19 '18 at 0:58
  • $\begingroup$ @JamesPhillips Does this mean I have to convert all the data with similar dimensions (say height) to the same unit and then rescale them? In this case, how does the comparison between height and weight play out? Does ensuring that weight is in grams and height in centimeters make any difference as compared to weight being in grams but height being in kilometers? $\endgroup$ – rahs Nov 19 '18 at 1:55
  • $\begingroup$ If you want to compare the magnitude of effect for the fitted coefficients, this is most easily done if all independent variables (IVs) have the same scale. The final model does not necessarily require such rescaling. You can make two models, one for analyzing magnitude of effect of the IVs, and if needed a second model, either using some rescaling or in the original units. $\endgroup$ – James Phillips Nov 19 '18 at 17:14
  • $\begingroup$ @JamesPhillips Alright, so to confirm, what you mean is that the coefficients may not be erroneous, but just look skewed because of possibly different dimensions used and that's that - there's nothing to be done.Is that correct? $\endgroup$ – rahs Nov 19 '18 at 22:30
  • $\begingroup$ This sounds similar to regressing fuel efficiency using "miles per hour" in the US with "kilometers per hour" in th EU for the same car - the physical relationship being modeled is the same, but the coefficient values are different. In such a case the coefficients are not erroneous, but the units are different. $\endgroup$ – James Phillips Nov 20 '18 at 0:20

Variable importance is not straightforward in linear regression since variables might be correlated.

However, its a good practice to do the following when preparing your data for training models:

  1. Ensure all variables representing the same quantities are in the same units (all distance in meters, weights in grams, or as per the scale suitable to the problem)
  2. Scale all variables to zero mean and unit standard deviation

Determining the importance of variables in linear regression

A popular approach to determine variable importance for linear regression models is to decompose the $R^2$ into contributions attributed to each variable.

Refer to the document describing the PMD method (Feldman, 2005). Another popular approach is averaging over orderings (LMG, 1980). The LMG works like this:

  • Find the semi-partial correlation of each predictor in the model, e.g. for variable a we have: $SS_a/SS_{total}$. It implies how much would $R^2$ increase if variable $a$ were added to the model.
  • Calculate this value for each variable for each order in which the variable gets introduced into the model, i.e. {$a,b,c$} ; {$b,a,c$} ; {$b,c,a$}
  • Find the average of the semi-partial correlations for each of these orders. This is the average over orderings.

The R package relaimpo implements 6 different metrics for assessing relative importance of variables in the linear regression model, including averaging over orderings of regressors and pmvd. relaimpo also gives bootstrap confidence intervals.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.