13
$\begingroup$

I am running a logistic model. The actual model dataset has more than 100 variables but I am choosing a test data set in which there are around 25 variables. Before that I also made a dataset which had 8-9 variables. I am being told that AIC and SC values can be used to compare the model. I observed that the model had higher SC values even when the variable had low p values ( ex. 0053) . To my intuition a model which has variables having good significance level should result in low SC and AIC values. But that isn't happening. Can someone please clarify this. In short I want to ask the following questions:

  1. Does the number of variable have anything to do with SC AIC ?
  2. Should I concentrate on p values or low SC AIC values ?
  3. What are the typical ways of reducing SC AIC values ?
$\endgroup$
15
$\begingroup$

It is quite difficult to answer your question in a precise manner, but it seems to me you are comparing two criteria (information criteria and p-value) that don't give the same information. For all information criteria (AIC, or Schwarz criterion), the smaller they are the better the fit of your model is (from a statistical perspective) as they reflect a trade-off between the lack of fit and the number of parameters in the model; for example, the Akaike criterion reads $-2\log(\ell)+2k$, where $k$ is the number of parameters. However, unlike AIC, SC is consistent: the probability of choosing incorrectly a bigger model converges to 0 as the sample size increases. They are used for comparing models, but you can well observe a model with significant predictors that provide poor fit (large residual deviance). If you can achieve a different model with a lower AIC, this is suggestive of a poor model. And, if your sample size is large, $p$-values can still be low which doesn't give much information about model fit. At least, look if the AIC shows a significant decrease when comparing the model with an intercept only and the model with covariates. However, if your interest lies in finding the best subset of predictors, you definitively have to look at methods for variable selection.

I would suggest to look at penalized regression, which allows to perform variable selection to avoid overfitting issues. This is discussed in Frank Harrell's Regression Modeling Strategies (p. 207 ff.), or Moons et al., Penalized maximum likelihood estimation to directly adjust diagnostic and prognostic prediction models for overoptimism: a clinical example, J Clin Epid (2004) 57(12).

See also the Design (lrm) and stepPlr (step.plr) R packages, or the penalized package. You may browse related questions on variable selection on this SE.

$\endgroup$
4
  • $\begingroup$ Hi chl,Thanksfor the reply..I admit that I got some information from your answer..Let me put my understanding and then you can comment please. (1) I get a hint that P values can go down if your sample size is big...--Is that so ?? To my understanding p values can only show whether or not ur null hypothesis is rejected. (2) I understand now that I need to see difference in AIC values with intercept only and with covariates. I suppose when we say that we want lower AIC we mean for the same dataset. I am getting character character left in my comment so will comment again once you answer please, $\endgroup$ Oct 26 '10 at 9:48
  • 1
    $\begingroup$ @ayush (1) the test statistics (e.g. Wald) depend on the sample size (the standard error decrease with increasing sample size, and you're likely to get lower p-values with a larger sample). (2) yes, although AIC may be used to compare non-nested models, here I was thinking of it as a way to compare different models of increasing complexity. $\endgroup$
    – chl
    Oct 26 '10 at 10:35
  • $\begingroup$ thanks again..I get the essence of the p value now. Some 5mins back I ran a model which is giving me p values below .05 for all the variables but AIC of 28238.407 with intercept only and with covariates 21507.933. I also have a case in which AIC is 16035.xy with intercept only and with covariates 4234.xy. What is your opinion comparing two cases ? Please note that the second model had different variables 25 var while first had 20. so second though had more variables ( 25 comparison to 20) had lower AIC. Though p values werent .05 for all. Please suggest..more to ask after this..Thanks. $\endgroup$ Oct 26 '10 at 11:33
  • $\begingroup$ @ayush It's difficult to answer about model quality without knowing how variables were selected. The gap in AIC between a model including only an intercept and some covariates gives you an indication about the "explanatory power" of those predictors (the residual deviance seems to decrease by a larger extent in the 2nd case you showed, and AIC penalizes for the # parameters as I said in my response). It's by no means a full answer about the relevance of these predictors. I'd suggest you to ask for a more specific question (IMO), e.g. about variable selection in GLMs for your specific study. $\endgroup$
    – chl
    Oct 26 '10 at 13:29
8
$\begingroup$

Grouping SC and AIC together IS WRONG. They are very different things, even though people heavily misuse them. AIC is meaningful when you are predicting things, using SC in this scenario can lead (not all the times) to wrong results. Similarly, if you are interested in doing model selection with the principle of parsimony (Occam's Razor) SC is better. I don't want to go into the theoretical details, but in a nutshell: SC -- good for parsimonious models when you want something equivalent to simplest possible model to explain your data, AIC -- When you want to predict. AIC doesn't assume that your true model lies in the model space where as SC does.

Secondly, using p-values and information criteria together can be also misleading as explained by chl.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.