How to understand and interpret multicollinearity in regression models

Question

I am using python to implement different regression models on a fantasy sports dataset. I am using a multivariable dataset which contains 5 independent variables to 2 regression models, which is Lasso from Sklearn and OLS from StatsModel. My implementation for both these models are as follows:

Lasso

• I am applying Lasso regression as the model can detect multicollinearity and thus reduce the variable coefficients to 0.
• I have normalised all dependent variables in the constructor method to ensure that the coefficient from the independent variables can be related to each other and have the same effect on the loss function.
• Value of R2 calculated using GridSearchCV where alpha value range is from 1e-3 to 10.

My results from Lasso model (1) show:

• Variables x1, x2 and x3 have very little effect on predicting the dependent variable (due to very low value of the coefficients = This indicates multicollinearity between them)
• VIF factors is greater than 5 for variable x1, x3 and x5
• Model gives a R2 score of 0.95446

My results from OLS model show:

• Variables x1 and x3 have been manually removed from the model as the VIF was greater than 5 (this also clears the condition number warning)
• Coefficient values for the remaining variables are close to the Lasso regression values
• Model gives a lower R2 score of 0.923076 (however I am more confident that multicollinearity does not influence the result)
• Note: I get the same results from the Lasso model if I remove x1 and x3 variables at the start just like the OLS method

I am struggling to understand the following:

• Should I manually check the VIF for all dependent variables and remove variables that have VIF greater than 5 before I start Lasso regression (as I then know that multicollinearity is not a problem in the model). Should the same approach be applied for the SVR Model in Sklearn aswell?

• What could be some of the reasons that are resulting in the difference between the R2 value between Lasso and OLS models?

• Is the Lasso model overfitting to the training data?

See attached images which show the result from the 2 models. Sorry for the long question (hopefully it is clear).

Lasso Results = VIF & Coefficients

OLS Results = VIF & Coefficients

OLS Summary Results

Answer 1

I can only answer this question in a confirmatory way. Normally, many people in machine learning have huge datasets with lots of features and rows or only a few features from Kaggle with a moderate amount of rows.

What is common to most people regardless of the dataset is that they do not derive a hypothesis or work out material. They see it as exploratory data analysis and want to confirm their opinion about something that could be in the data. Sometimes this can be appropriate, but normally you would derive a hypothesis. Thus, when you derive a theoretical construct about your use-case and what you are doing you should normally have considered multicollinearity of some features, especially in a case of regression where you first look at correlations. Thus, I would see it, from a confirmatory way, you should exclude the variables before doing your regression.

I see the other regression methods more like a fallback. Imagine you have 5000 features in tabular data. You can not check for multicollinearity for all of them. You want your algorithm to deal with that by some sort of lowering the impact. But this can not be as good as excluding variables upfront.

BTW. if the methods (Lasso, Linear) weren't executed on a ML pipeline, so that the cross validation samples are all the same for both regression tasks, and thus comparable, you cant compare both $R^2$.

Hope that helps you.

Answer 2

I agree with Patrick's comments above.

I found the following articles useful which highlight that removing independent variables related to multicollinearity will improve the model output and this can be performed using a loop.

• https://beckmw.wordpress.com/2013/02/05/collinearity-and-stepwise-vif-selection/
• Why is multicollinearity not checked in modern statistics/machine learning
• https://www.kaggle.com/robertoruiz/dealing-with-multicollinearity

The VIF factor can be checked automatically by creating a loop. See my sample code below.

Example code:

from statsmodels.stats.outliers_influence import variance_inflation_factor
import statsmodels.api as sm

def check_vif(self):
    
    def_vif_X = self.def_X.copy()
    def_vif_X = sm.add_constant(def_vif_X)
    def_columns_removed = []
    
    def_vif_check = pd.DataFrame()
    def_vif_check["VIF"] = [variance_inflation_factor(def_vif_X.values, i) for i in range(def_vif_X.shape[1])]
    def_vif_check["features"] = def_vif_X.columns

    def_vif_check_condition = False
    
    while def_vif_check_condition == False:
        
        vif_const_index = (def_vif_check[def_vif_check["features"] == "const"]).index
        def_vif_check.drop(vif_const_index, axis=0, inplace=True)
        def_vif_check.sort_values(by="VIF", ascending=False, inplace=True)
        def_vif_check.reset_index(drop=True, inplace=True)
        vif_number = def_vif_check.loc[0, "VIF"]
        
        if vif_number < 5:
            def_vif_check_condition = True
            break
        else:
            def_columns_removed.append(def_vif_check.loc[0, "features"])
            def_col_removed_last = def_columns_removed[len(def_columns_removed) - 1]
            def_vif_X.drop(def_col_removed_last, axis=1, inplace=True)
            def_vif_check = pd.DataFrame()
            def_vif_check["VIF"] = [variance_inflation_factor(def_vif_X.values, i) for i in range(def_vif_X.shape[1])]
            def_vif_check["features"] = def_vif_X.columns