6
$\begingroup$

I'm designing large-scale, regularized logistic regression models with lots of sparse, binarized features. e.g. isUS, isFR, etc. As a result, a lot of the weights in the model are zero.

I'm wondering how I should compute the "number of parameters" in model-selection criteria like AIC, BIC, etc. Should I only count the number of non-zero weights or all of the weights?

e.g. If there are 10 possible countries, but only 6 have non-zero weights, is the number of parameters 6 or 10?

$\endgroup$
  • $\begingroup$ are you trying to build your own AIC measurement? Are you using software that doesn't include AIC for some reason? $\endgroup$ – Andrew Cassidy Mar 10 '14 at 20:21
7
$\begingroup$

Degrees of freedom do not depend on the outcome alone but on the fitting procedure. If it's maximum likelihood, all parameters count.

There is an interesting case where zero weights do not count, and that's lasso: H Zou, T Hastie, R Tibshirani On the “degrees of freedom” of the lasso. The Annals of Statistics, 2007

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ You're right. I was thinking of a lasso context. In general, for regularized regression df will be less than p and will vary as a function of the amount of regularization. I deleted my incorrect answer. $\endgroup$ – Jeremy Coyle Mar 11 '14 at 20:32
  • $\begingroup$ So when using L1 regularization, would you only count non-zero weights? $\endgroup$ – tskuzzy Mar 12 '14 at 18:21
6
+150
$\begingroup$

This is a really difficult question to answer without precise knowledge of the fitting algorithm, nor is it clear cut that there is a reasonable definition of the "number of parameters" that will justify AIC, BIC or other "information criteria" in general.

If estimation is done by $\ell_1$-penalized maximum-likelihood estimation, then I can partially iterate the answer by user27493. In this case the estimated number of non-zero parameters is a sensible substitute for the total number of parameters in AIC. Note, however, that the Zou et al. paper is on least squares regression with an $\ell_1$-penalty $-$ not logistic regression. See, for instance, Differential geometric least angle regression: a differential geometric approach to sparse generalized linear models by L. Augugliaro et al. for results related to generalized linear models.

BIC is different, and I don't know results in this direction.

The paper with the catchy title Effective Degrees of Freedom: A Flawed Metaphor, posted recently on archive by Lucas Janson, William Fithian, and Trevor Hastie, shows that, depending on the data generating mechanism, the effective degrees of freedom ("number of parameters") may exceed the total number of parameters, and may even be unbounded.

In this paper (shameless self promotion of my research) Degrees of freedom for nonlinear least squares estimation with my coauthor Alexander Sokol, we show that for nonlinear least squares estimation the effective degrees of freedom generally contains a hard-to-estimate term that depends on the data generating model. This is also what pops up in some of the examples in the Janson et al. paper mentioned above. In an asymptotic scenario, if the model is close to being true and/or if the model does not "curve too much", and if you use $\ell_1$-penalized least squares estimation, a useful surrogate estimate of the effective degrees of freedom is still the estimated number of non-zero parameters. However, once you move outside of some of the standard and most well behaved models, anything could happen.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.