I am running a regression model both with Lasso and Ridge (to predict a discrete outcome variable ranging from 0-5). Before running the model, I use
SelectKBest method of
scikit-learn to reduce the feature set from 250 to 25. Without an initial feature selection, both Lasso and Ridge yield to lower accuracy scores [which might be due to the small sample size, 600]. Also, note that some features are correlated.
After running the model, I observe that the prediction accuracy is almost the same with Lasso and Ridge. However, when I check first 10 features after ordering them by the absolute value of coefficients, I see that there is at most %50 overlap.
That is, given that different importance of features were assigned by each method, I might have a totally different interpretationbased on the model I choose.
Normally, the features represent some aspects of user behavior in a web site. Therefore, I want to explain the findings by highlighting the features (user behaviors) with stronger predictive ability vs weaker features (user behaviors). However, I do not know how to move forward at this point. How should I approach to interpreting the model? For example, should combine both and highlight the overlapping one, or should I go with Lasso since it provides more interpretability?