I am using python to implement different regression models on a fantasy sports dataset. I am using a multivariable dataset which contains 5 independent variables to 2 regression models, which is Lasso from Sklearn and OLS from StatsModel. My implementation for both these models are as follows:


  • I am applying Lasso regression as the model can detect multicollinearity and thus reduce the variable coefficients to 0.
  • I have normalised all dependent variables in the constructor method to ensure that the coefficient from the independent variables can be related to each other and have the same effect on the loss function.
  • Value of R2 calculated using GridSearchCV where alpha value range is from 1e-3 to 10.

My results from Lasso model (1) show:

  • Variables x1, x2 and x3 have very little effect on predicting the dependent variable (due to very low value of the coefficients = This indicates multicollinearity between them)
  • VIF factors is greater than 5 for variable x1, x3 and x5
  • Model gives a R2 score of 0.95446

My results from OLS model show:

  • Variables x1 and x3 have been manually removed from the model as the VIF was greater than 5 (this also clears the condition number warning)
  • Coefficient values for the remaining variables are close to the Lasso regression values
  • Model gives a lower R2 score of 0.923076 (however I am more confident that multicollinearity does not influence the result)
  • Note: I get the same results from the Lasso model if I remove x1 and x3 variables at the start just like the OLS method

I am struggling to understand the following:

  • Should I manually check the VIF for all dependent variables and remove variables that have VIF greater than 5 before I start Lasso regression (as I then know that multicollinearity is not a problem in the model). Should the same approach be applied for the SVR Model in Sklearn aswell?

  • What could be some of the reasons that are resulting in the difference between the R2 value between Lasso and OLS models?

  • Is the Lasso model overfitting to the training data?

See attached images which show the result from the 2 models. Sorry for the long question (hopefully it is clear).

Lasso Results = VIF & Coefficients enter image description here

OLS Results = VIF & Coefficients enter image description here

OLS Summary Results enter image description here

enter image description here

enter image description here


2 Answers 2


I can only answer this question in a confirmatory way. Normally, many people in machine learning have huge datasets with lots of features and rows or only a few features from Kaggle with a moderate amount of rows.

What is common to most people regardless of the dataset is that they do not derive a hypothesis or work out material. They see it as exploratory data analysis and want to confirm their opinion about something that could be in the data. Sometimes this can be appropriate, but normally you would derive a hypothesis. Thus, when you derive a theoretical construct about your use-case and what you are doing you should normally have considered multicollinearity of some features, especially in a case of regression where you first look at correlations. Thus, I would see it, from a confirmatory way, you should exclude the variables before doing your regression.

I see the other regression methods more like a fallback. Imagine you have 5000 features in tabular data. You can not check for multicollinearity for all of them. You want your algorithm to deal with that by some sort of lowering the impact. But this can not be as good as excluding variables upfront.

BTW. if the methods (Lasso, Linear) weren't executed on a ML pipeline, so that the cross validation samples are all the same for both regression tasks, and thus comparable, you cant compare both $R^2$.

Hope that helps you.

  • $\begingroup$ Thanks for the comment Patrick. I agree that removing multicollinearity before completing any regression will provide better results and more robust model (I saw better results from Lasso regression model after removing multicollinearity independent variables). In addition, I googled some more over the last few days and found examples where the VIF factor is calculated in a loop and multicollinearity is checked for all variables. I have followed the same approach in my project. Lastly, I going to update my question with an answer below which may help anyone else. $\endgroup$
    – tb08
    Commented Mar 4, 2021 at 22:30

I agree with Patrick's comments above.

I found the following articles useful which highlight that removing independent variables related to multicollinearity will improve the model output and this can be performed using a loop.

The VIF factor can be checked automatically by creating a loop. See my sample code below.

Example code:

from statsmodels.stats.outliers_influence import variance_inflation_factor
import statsmodels.api as sm

def check_vif(self):
    def_vif_X = self.def_X.copy()
    def_vif_X = sm.add_constant(def_vif_X)
    def_columns_removed = []
    def_vif_check = pd.DataFrame()
    def_vif_check["VIF"] = [variance_inflation_factor(def_vif_X.values, i) for i in range(def_vif_X.shape[1])]
    def_vif_check["features"] = def_vif_X.columns

    def_vif_check_condition = False
    while def_vif_check_condition == False:
        vif_const_index = (def_vif_check[def_vif_check["features"] == "const"]).index
        def_vif_check.drop(vif_const_index, axis=0, inplace=True)
        def_vif_check.sort_values(by="VIF", ascending=False, inplace=True)
        def_vif_check.reset_index(drop=True, inplace=True)
        vif_number = def_vif_check.loc[0, "VIF"]
        if vif_number < 5:
            def_vif_check_condition = True
            def_columns_removed.append(def_vif_check.loc[0, "features"])
            def_col_removed_last = def_columns_removed[len(def_columns_removed) - 1]
            def_vif_X.drop(def_col_removed_last, axis=1, inplace=True)
            def_vif_check = pd.DataFrame()
            def_vif_check["VIF"] = [variance_inflation_factor(def_vif_X.values, i) for i in range(def_vif_X.shape[1])]
            def_vif_check["features"] = def_vif_X.columns
  • $\begingroup$ Gave you an upvote for this! :-) $\endgroup$ Commented Mar 5, 2021 at 8:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.