I'm currently running a ridge regression in R using the glmnet
package, however, I recently ran into a new problem and was hoping for some help in interpreting my results. My data can be found here: https://www.dropbox.com/sh/hpxu3t0vqkrzfgf/AAB6F-yMYMfuI5E__gfDuW6sa?dl=0
My data consists of a 26531x428 observation matrix x
and a 26531x1 response vector y
. I am attempting to determine the optimal value of lambda.min
, and when I run the code
> lambda=cv.glmnet(x=x,y=y,weights=weights,alpha=0,nfolds=10,standardize=FALSE)
I get
$lambda.min
[1] 2.123479
$lambda.1se
[1] 619.0054
which are results I would expect. However, I would like to add a slight tweak to this regression. I have prior knowledge of each of my 428 coefficients, and instead of shrinking each coefficient towards 0, as is the default with ridge regression, I would like to shrink each coefficient towards a specific value other than 0. After reaching out to Dr. Trevor Hastie, one of the creators of glmnet
, he told me that this could be achieved by running the same code after substituting y
with y2
, where y2 = y - x%*%d
and d
is a 428x1 vector of coefficient priors. He said to then add d
to my new coefficients, which would give me my prior-informed coefficients. After rerunning the code
> lambda=cv.glmnet(x=x,y=y2,weights=weights,alpha=0,nfolds=10,standardize=FALSE)
I unfortunately get
$lambda.min
[1] 220.3026
$lambda.1se
[1] 220.3026
The results of plot(lambda)
look like this
Does anyone know why glmnet
can't find a suitable lambda.min
? Could it be because my vector of priors contains estimates that are too far off? Any help would be greatly appreciated!