2
$\begingroup$

I'm working on a high-ish dimensional logistic regression problem. I have 40 variables ($p=40$) and 900 samples ($n=900$), but only 30 of those samples are in the class I'm trying to predict.

I'm fitting a predictive model using penalized maximum likelihood (ridge regression, via glmnet). This requires setting a value for the penalty term $\lambda$. Typically, $\lambda$ is determined through cross-validation.

I've also been reading Frank Harrell, who suggests keeping a model's degrees of freedom below a fraction of the "limiting sample size", $m$. As a rule of thumb, he suggests $df < m/15$. In my case, where $m = 30$, this would mean keeping the degrees of freedom below 2.

The effective degrees of freedom in ridge regression can be calculated as a function of $\lambda$ , $$ df(\lambda) = \sum^{p}_{j=1} d_j^2 / (d_j^2 + \lambda) $$ where $d_j$'s are the singular values of the sample matrix (ESL, eq 3.50).

In ridge regression (or other penalized maximum likelihood techniques), is it ever advisable to choose $\lambda$ using the limiting sample size $m$, instead of choosing it through cross-validation? When might this be wise?

$\endgroup$
2
$\begingroup$

Harrell himself is a user here and might chime in. Failing that, and not having a copy of the book in question, I can only say that I see no reason to use degrees of freedom and $m$ to choose $λ$. The motivation for using cross-validation is that you want to choose the $λ$ that maximizes predictive accuracy, and cross-validation gives a good estimate of predictive accuracy. I don't see why using $m$ would get predictive accuracy that's any better.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.