glmnet for binary outcomes: Why is "%Dev" inversely correlated with lambda?

Question

I am new to glmnet but would like to apply it to a dataset with binary outcomes. Can you please clarify a few questions for me? Below are the codes and data setup

data(BinomialExample)
set.seed(33)
head(x) 
head(y)
cvfit <- cv.glmnet(x, y, type.measure="deviance",alpha=0.5, family="binomial")

(1) In the output below, what is the %Dev” and why is %Dev negatively correlated with $\lambda$?

Specifically, the glmnet vignette said %Dev is the percentage of deviance explained—but on which set? Is it on each holdout set, where %Dev is calculated by running prediction (on the holdout set) with the coefficients generated from training set?

> cvfit$glmnet.fit

Call:  glmnet(x = x, y = y, alpha = 0.5, family = "binomial") 

    Df    %Dev  Lambda
1    0 0.00000 0.48100
2    1 0.01571 0.43820
3    2 0.03147 0.39930
4    2 0.05559 0.36380
5    2 0.07810 0.33150
6    2 0.09907 0.30210
7    3 0.11930 0.27520

plot(cvfit$lambda,cvfit$glmnet.fit$dev.ratio)

(2) In the plot below, how is the “binomial deviance” (y-axis) corresponding to the optimal $\lambda$ (dash lines) calculated? And how is this “binomial deviance” different from those reported in %Dev as described in question (1)?

I understand that the optimal lambda is chosen to minimize the (cross-validated) deviance. Is the “binomial deviance” shown below defined as averaging over the deviance at fold 1,2…10 (assuming 10-fold CV), where at each fold the deviance is calculated by predicting on the holdout set, using coefficients obtained from training?

plot(cvfit)

(3) On which set is the null deviance (cvfit$glmnet.fit$nulldev) defined?

It looks like cvfit$glmnet.fit$nulldev is a constant; is it the null deviance defined on the overall sample?

If glment has %Dev defined on individual holdout set, wouldn’t it be more convenient to have the null deviance for each holdout set (so that, for example, one can calculate pseudo R-squared at each holdout set), instead reporting null deviance for the overall?

> cvfit$glmnet.fit$nulldev
[1] 137.186

Answer 1

Although this seems to be primarily a question about code in a specific package (off topic on this site), there is enough general statistical content to address briefly.

Most of your general questions about deviance in cross-validation of a binomial/logistic model are answered quite well on this page. There is no fundamental difference from using cross-validation for an ordinary least squares model: what you evaluate is the performance averaged over the hold-out sets. The performance in this case is gauged by the deviance. Open source code makes it fairly easy to get as deep as you want into the implementation. In this case, the trick is to recognize that the object produced by glmnet() for a binomial model is of class "lognet" so that the work is done by the function cv.lognet(). The code for that function will be printed if the package is loaded and you type cv.lognet at the R prompt.

With respect to the specific questions:

1. The manual page for cv.glmnet specifies that the $glmnet.fit component of its returned object is "a fitted glmnet object for the full data." As you increase the penalty $\lambda$ the increasing penalization on the coefficient magnitudes necessarily means that the fitted model will match the data less closely and explain less of the deviance.

2. As already noted, performance is evaluated on the held-out data in the cross-validation. Your second plot is based on the cross-validated averaged deviance values themselves, not the deviance ratio as in your first question and first plot; see the page linked above for the normalization apparently applied to the values.

3. As noted in the answer to question (1) the $glmnet.fit component is for the full data, so its sub-component $nulldev will also be for the full data. One could adapt the code to report details for individual folds for each value of $\lambda$, but it's not clear what practical advantage using pseudo R-squared values would provide over the deviance for finding an optimal value of $\lambda$ (whether that "optimal" is defined as the $\lambda$ value giving the minimum or the largest $\lambda$ within 1 standard error of the minimum.).

Answer 2

Thanks @EdM for the clarification! Now I can appreciate the difference between deviance defined on the full data (question 1) and cross-validated averaged deviance (question 2).

The page you mentioned provides great explanation on how deviance is defined on an glment object, as well as the apparent normalization applied to the cross-validated deviance.

To illustrate the relationship between the two different operational definitions for deviance, I generated the plot:

plot(cvfit$cvm, deviance(cvfit$glmnet.fit)/length(y),main="cross-validated deviance (x-axis) and {full-data deviance/100} (y-axis)")

The plot shows that there may be some kind of relationship between the two, which is probably what the "normalization" mentioned in this page was about.