Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I'm designing large-scale, regularized logistic regression models with lots of sparse, binarized features. e.g. isUS, isFR, etc. As a result, a lot of the weights in the model are zero.
I'm wondering how I should compute the "number of parameters" in model-selection criteria like AIC, BIC, etc. Should I only count the number of non-zero weights or all of the weights?
e.g. If there are 10 possible countries, but only 6 have non-zero weights, is the number of parameters 6 or 10?
Degrees of freedom do not depend on the outcome alone but on the fitting procedure. If it's maximum likelihood, all parameters count.
There is an interesting case where zero weights do not count, and that's lasso: H Zou, T Hastie, R Tibshirani On the “degrees of freedom” of the lasso. The Annals of Statistics, 2007
This is a really difficult question to answer without precise knowledge of the fitting algorithm, nor is it clear cut that there is a reasonable definition of the "number of parameters" that will justify AIC, BIC or other "information criteria" in general.
If estimation is done by $\ell_1$-penalized maximum-likelihood estimation, then I can partially iterate the answer by user27493. In this case the estimated number of non-zero parameters is a sensible substitute for the total number of parameters in AIC. Note, however, that the Zou et al. paper is on least squares regression with an $\ell_1$-penalty $-$ not logistic regression. See, for instance, Differential geometric least angle regression: a differential geometric approach to sparse generalized linear models by L. Augugliaro et al. for results related to generalized linear models.
BIC is different, and I don't know results in this direction.
The paper with the catchy title Effective Degrees of Freedom: A Flawed Metaphor, posted recently on archive by Lucas Janson, William Fithian, and Trevor Hastie, shows that, depending on the data generating mechanism, the effective degrees of freedom ("number of parameters") may exceed the total number of parameters, and may even be unbounded.
In this paper (shameless self promotion of my research) Degrees of freedom for nonlinear least squares estimation with my coauthor Alexander Sokol, we show that for nonlinear least squares estimation the effective degrees of freedom generally contains a hard-to-estimate term that depends on the data generating model. This is also what pops up in some of the examples in the Janson et al. paper mentioned above. In an asymptotic scenario, if the model is close to being true and/or if the model does not "curve too much", and if you use $\ell_1$-penalized least squares estimation, a useful surrogate estimate of the effective degrees of freedom is still the estimated number of non-zero parameters. However, once you move outside of some of the standard and most well behaved models, anything could happen.
