Understanding AIC and Schwarz criterion

Question

I am running a logistic model. The actual model dataset has more than 100 variables but I am choosing a test data set in which there are around 25 variables. Before that I also made a dataset which had 8-9 variables. I am being told that AIC and SC values can be used to compare the model. I observed that the model had higher SC values even when the variable had low p values ( ex. 0053) . To my intuition a model which has variables having good significance level should result in low SC and AIC values. But that isn't happening. Can someone please clarify this. In short I want to ask the following questions:

1. Does the number of variable have anything to do with SC AIC ?
2. Should I concentrate on p values or low SC AIC values ?
3. What are the typical ways of reducing SC AIC values ?

Answer 1

It is quite difficult to answer your question in a precise manner, but it seems to me you are comparing two criteria (information criteria and p-value) that don't give the same information. For all information criteria (AIC, or Schwarz criterion), the smaller they are the better the fit of your model is (from a statistical perspective) as they reflect a trade-off between the lack of fit and the number of parameters in the model; for example, the Akaike criterion reads $-2\log(\ell)+2k$, where $k$ is the number of parameters. However, unlike AIC, SC is consistent: the probability of choosing incorrectly a bigger model converges to 0 as the sample size increases. They are used for comparing models, but you can well observe a model with significant predictors that provide poor fit (large residual deviance). If you can achieve a different model with a lower AIC, this is suggestive of a poor model. And, if your sample size is large, $p$-values can still be low which doesn't give much information about model fit. At least, look if the AIC shows a significant decrease when comparing the model with an intercept only and the model with covariates. However, if your interest lies in finding the best subset of predictors, you definitively have to look at methods for variable selection.

I would suggest to look at penalized regression, which allows to perform variable selection to avoid overfitting issues. This is discussed in Frank Harrell's Regression Modeling Strategies (p. 207 ff.), or Moons et al., Penalized maximum likelihood estimation to directly adjust diagnostic and prognostic prediction models for overoptimism: a clinical example, J Clin Epid (2004) 57(12).

See also the Design (lrm) and stepPlr (step.plr) R packages, or the penalized package. You may browse related questions on variable selection on this SE.

Answer 2

Grouping SC and AIC together IS WRONG. They are very different things, even though people heavily misuse them. AIC is meaningful when you are predicting things, using SC in this scenario can lead (not all the times) to wrong results. Similarly, if you are interested in doing model selection with the principle of parsimony (Occam's Razor) SC is better. I don't want to go into the theoretical details, but in a nutshell: SC -- good for parsimonious models when you want something equivalent to simplest possible model to explain your data, AIC -- When you want to predict. AIC doesn't assume that your true model lies in the model space where as SC does.

Secondly, using p-values and information criteria together can be also misleading as explained by chl.