2
$\begingroup$

I am selecting a GLM model from a large data set based on an optimal AIC (Akaike information criterion ) for a set of candidate models. There about ~50 categorical factors in my model, each with several levels. I want to gain some information whether a specific factor is significant based on AIC value. In order to do this, I build 2 models in a forward stepwise fashion: model 1 with $P$ base factors included and then model 2 with the addition of a candidate factor, overall $P+1$ factors. The quantity of an interest is the difference in AIC value.

For example, using a GLM with log-link and Poisson error structure for some simulated data, I find in a base model that the AIC is equal to $160000$. If I include a factor which is predictive, the change in AIC is $\approx 1000$. If I include a spurious factor the change is $\approx 100$. I figure we can empirically conclude that for a predictive factor, the change should be $\geq 1000$.

My question is how theoretically calculate or estimate the minimal value of change in AIC value, sufficient to consider the factor to be significant for our GLM model? I also want to know if there is any similar result concerning Deviance.

$\endgroup$
  • $\begingroup$ 1. The AIC is $2K - 2 \log \left( \mathcal{L} \right)$, so lower is better. 2. I hope the 1,000 unit difference is a negative difference favoring the $P+1$ model. 3. It's worrying you see a 100 unit difference after adding a spurious factor if it the same direction. 4. Theoretically, the AIC will be greater (not less) by adding a spurious factor. Forward stepwise model selection with AIC would recommend you include a factor if it has a difference of 0 or less. By all means, you can set a threshold to what you want. The AIC overfits. $\endgroup$ – AdamO Apr 9 '18 at 21:18
1
$\begingroup$

I cannot imagine how you would use the AIC for what you are intending to do as it has no concept of significance connected to it.

There are two ways you could view the AIC, BIC or DIC. First, you could take them as being derived from information theory. From an information theoretic perspective, the differences are differences in information. You choose the one with the greatest amount of unique information. This has nothing at all to do with statistical significance. A variable could be required to be present by the AIC, but not statistically significant by the t-test. You still cannot remove it because there will be information loss. The failure to test significance simply means you cannot show to a high degree of confidence that the variable does not belong in the regression, it doesn't mean it doesn't belong in the regression.

The second view would be as a stylized approximation of the Bayesian posterior density function. In this view, the value with the smallest AIC is the one with the greatest probability of being the data generating function in nature. The differences in the AIC could then be thought of as being an approximation of the unnormalized posterior probability between two choices. In that case, a 1 unit difference would vary by $\frac{p}{1-p}=e^1$>. Given a pairwise forced comparison, the higher model has a 73% chance of being correct and the other a 27% chance of being correct. This isn't valid because there are many models and the probabilities have not been normalized. The odds are valid, but they are relative odds and potentially crude approximations of the true posterior. You cannot get to Frequentist probabilities from here. The various information criterion also makes distribution assumptions that are suspect here.

The other problem is that you are assuming that simply adding or subtracting variables gives you a linear relationship either among the probabilities or the information loss. It may be that model 105 is improved in terms of AIC by adding variable $x$, but that model 212 is improved by removing variable $x$.

You are missing an information theoretic issue too. Bayesian methods only incorporate unique information once. If unique information was captured in some set then a new set of information would only capture the information not included in the first set. The posterior improves only with the addition of marginal information, not with the addition of data. This is approximated with the $2\ln(\hat{L})$ term. There is a possibility of a spurious linkage of information becoming mis-included because the entire density is being left out. That is the price of using the approximation rather than the exact posterior, but you gain enormous calculation speed.

Because the Bayesian posterior only updates on the addition of unique information, there is no simple way, if a way exists, to extract the model specification from the particular sample's peculiarities. To give an example, let us imagine variable $x$ is not really in the true data generating function, but due to getting an unusual sample and due to $x$ being correlated with $y$ which is in the data generating function, we sometimes find that $x$ shows up as being the correct model rather than $y$. This is due to the fact that in some weird configurations of data the information content of $y$ is much less informative than $x$, though over the sample space this is rarely true.

Also, for your specific case, you may want to consider the DIC for a multi-level model.

For your simulation, you just want the smallest AIC unless the distance is really trivial. If you have added spurious data streams and find the AIC is moving in the wrong direction, it could be that you have accidentally created spurious correlations in the structure of the data. It is also possible that the AIC is badly misspecified. The AIC and the BIC have broad general assumptions that are usually met, but can easily be inappropriate for your specific problem.

You should read the derivation of the AIC to decide if it is correct for your problem.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.