I am using python to implement different regression models on a fantasy sports dataset. I am using a multivariable dataset which contains 5 independent variables to 2 regression models, which is Lasso from Sklearn and OLS from StatsModel. My implementation for both these models are as follows:
- I am applying Lasso regression as the model can detect multicollinearity and thus reduce the variable coefficients to 0.
- I have normalised all dependent variables in the constructor method to ensure that the coefficient from the independent variables can be related to each other and have the same effect on the loss function.
- Value of R2 calculated using GridSearchCV where alpha value range is from 1e-3 to 10.
My results from Lasso model (1) show:
- Variables x1, x2 and x3 have very little effect on predicting the dependent variable (due to very low value of the coefficients = This indicates multicollinearity between them)
- VIF factors is greater than 5 for variable x1, x3 and x5
- Model gives a R2 score of 0.95446
My results from OLS model show:
- Variables x1 and x3 have been manually removed from the model as the VIF was greater than 5 (this also clears the condition number warning)
- Coefficient values for the remaining variables are close to the Lasso regression values
- Model gives a lower R2 score of 0.923076 (however I am more confident that multicollinearity does not influence the result)
- Note: I get the same results from the Lasso model if I remove x1 and x3 variables at the start just like the OLS method
I am struggling to understand the following:
Should I manually check the VIF for all dependent variables and remove variables that have VIF greater than 5 before I start Lasso regression (as I then know that multicollinearity is not a problem in the model). Should the same approach be applied for the SVR Model in Sklearn aswell?
What could be some of the reasons that are resulting in the difference between the R2 value between Lasso and OLS models?
Is the Lasso model overfitting to the training data?
See attached images which show the result from the 2 models. Sorry for the long question (hopefully it is clear).
Lasso Results = VIF & Coefficients
OLS Results = VIF & Coefficients