I am going through the LAB section §6.6 on Ridge Regression/Lasso in the book 'An Introduction to Statistical Learning with Applications in R' by James, Witten, Hastie, Tibshirani (2013).
More specifically, I am trying to do apply the scikit-learn Ridge
model to the 'Hitters' dataset from the R package 'ISLR'. I have created the same set of features as shown in the R code. However, I cannot get close to the results from the glmnet()
model. I have selected one L2 tuning parameter to compare. ('alpha' argument in scikit-learn).
Python:
regr = Ridge(alpha=11498)
regr.fit(X, y)
http://nbviewer.ipython.org/github/JWarmenhoven/ISL-python/blob/master/Notebooks/Chapter%206.ipynb
R:
Note that the argument alpha=0
in glmnet()
means that a L2 penalty should be applied (Ridge regression). The documentation warns not to enter a single value for lambda
, but the result is the same as in ISL, where a vector is used.
ridge.mod <- glmnet(x,y,alpha=0,lambda=11498)
What causes the differences?
Edit:
When using penalized()
from the penalized package in R, the coefficients are the same as with scikit-learn.
ridge.mod2 <- penalized(y,x,lambda2=11498)
Maybe the question could then also be: 'What is the difference between glmnet()
and penalized()
when doing Ridge regression?
New python wrapper for actual Fortran code used in R package glmnet
https://github.com/civisanalytics/python-glmnet
sklearn.linear_model.Ridge
does unpenalized intercept estimation (standard) and the penalty is such that||Xb - y - intercept||^2 + alpha ||b||^2
is minimized forb
. There can be factors1/2
or1/n_samples
or both in front of the penalty, making results different immediately. To factor out the penalty scaling problem, set the penalty to 0 in both cases, resolve any discrepancies there and then check what adding back the penalty does. And btw IMHO here IS the right place to ask this question. $\endgroup$