Questions tagged [aic]

AIC stands for the Akaike Information Criterion, which is one technique used to select the best model from a class of models using a penalized likelihood. A smaller AIC implies a better model.

Filter by
Sorted by
Tagged with
221
votes
9answers
102k views

Algorithms for automatic model selection

I would like to implement an algorithm for automatic model selection. I am thinking of doing stepwise regression but anything will do (it has to be based on linear regressions though). My problem ...
251
votes
12answers
174k views

Is there any reason to prefer the AIC or BIC over the other?

The AIC and BIC are both methods of assessing model fit penalized for the number of estimated parameters. As I understand it, BIC penalizes models more for free parameters than does AIC. Beyond a ...
72
votes
7answers
44k views

Do all interactions terms need their individual terms in regression model?

I am actually reviewing a manuscript where the authors compare 5-6 logit regression models with AIC. However, some of the models have interaction terms without including the individual covariate terms....
19
votes
3answers
19k views

Comparing non nested models with AIC

Say we have to GLMMs mod1 <- glmer(y ~ x + A + (1|g), data = dat) mod2 <- glmer(y ~ x + B + (1|g), data = dat) These models are not nested in the usual ...
27
votes
3answers
18k views

Prerequisites for AIC model comparison

What are exactly the prerequisites, that need to be fulfilled for AIC model comparison to work? I just came around this question when I did comparison like this: ...
14
votes
2answers
12k views

Non-nested model selection

Both the likelihood ratio test and the AIC are tools for choosing between two models and both are based on the log-likelihood. But, why the likelihood ratio test can't be used to choose between two ...
80
votes
5answers
75k views

What do the residuals in a logistic regression mean?

In answering this question John Christie suggested that the fit of logistic regression models should be assessed by evaluating the residuals. I'm familiar with how to interpret residuals in OLS, they ...
23
votes
3answers
5k views

Paradox in model selection (AIC, BIC, to explain or to predict?)

Having read Galit Shmueli's "To Explain or to Predict" (2010) and some literature on model selection using AIC and BIC, I am puzzled by an apparent contradiction. There are three premises, AIC- ...
12
votes
2answers
15k views

Testing the difference in AIC of two non-nested models

The whole point of AIC or any other information criterion is that less is better. So if I have two models M1: y = a0 + XA + e and M2: y = b0 + ZB + u, and if the AIC of the first (A1) is less than ...
37
votes
3answers
21k views

Logistic Regression: Bernoulli vs. Binomial Response Variables

I want to perform logistic regression with the following binomial response and with $X_1$ and $X_2$ as my predictors. I can present the same data as Bernoulli responses in the following format. ...
24
votes
3answers
93k views

AIC or p-value: which one to choose for model selection?

I'm brand new to this R thing but am unsure which model to select. I did a stepwise forward regression selecting each variable based on the lowest AIC. I came up with 3 models that I'm unsure which ...
33
votes
3answers
11k views

Can AIC compare across different types of model?

I'm using AIC (Akaike's Information Criterion) to compare non-linear models in R. Is it valid to compare the AICs of different types of model? Specifically, I'm comparing a model fitted by glm versus ...
13
votes
1answer
2k views

Can you compare AIC values as long as the models are based on the same dataset?

I am doing some forecasting in R using Rob Hyndman's forecast package. The paper belonging to the package can be found here. In the paper, after explaining the automatic forecasting algorithms, the ...
8
votes
1answer
14k views

AIC versus Likelihood Ratio Test in Model Variable Selection

The software that I am currently using to build a model compares a "current run" model to a "reference model" and reports (where applicable) both a chi-squared p-value based on likelihood ratio tests ...
3
votes
2answers
4k views

AIC/BIC and data transformation

Can you use AIC/BIC to compare models on untransformed data with models on transformed data (such as log, inverse hyperbolic sine, etc.)? I.e. if a model using logged data gives an AIC = 53.62 and a ...
10
votes
1answer
2k views

Equivalence of AIC and p-values in model selection

In a comment to the answer of this question, it was stated that using AIC in model selection was equivalent to using a p-value of 0.154. I tried it in R, where I used a "backward" subset selection ...
12
votes
2answers
13k views

REML vs ML stepAIC

I feel overwhelmed after attempting to dig into the literature on how to run my mixed model analysis following it up with using AIC to select the best model or models. I do not think my data is that ...
13
votes
3answers
10k views

When is it appropriate to select models by minimising the AIC?

It is well established, at least among statisticians of some higher calibre, that models with the values of the AIC statistic within a certain threshold of the minimum value should be considered as ...
5
votes
2answers
1k views

What is the actual significance of a difference in AIC or BIC values?

Usually, when a difference of a statistic is discussed, that discussion is presented in the context of a significance of that difference. When self-entropy, i.e., information content, is examined, ...
30
votes
3answers
46k views

What is the difference in what AIC and c-statistic (AUC) actually measure for model fit?

Akaike Information Criterion (AIC) and the c-statistic (area under ROC curve) are two measures of model fit for logistic regression. I am having trouble explaining what is going on when the results of ...
40
votes
5answers
198k views

AIC guidelines in model selection

I typically use BIC as my understanding is that it values parsimony more strongly than does AIC. However, I have decided to use a more comprehensive approach now and would like to use AIC as well. I ...
36
votes
3answers
7k views

What does the Akaike Information Criterion (AIC) score of a model mean?

I have seen some questions here about what it means in layman terms, but these are too layman for for my purpose here. I am trying to mathematically understand what does the AIC score mean. But at ...
9
votes
1answer
4k views

On Negative AIC Values

My question is related to the thread Negative values for AIC in General Mixed Model. I often get negative AIC values from the software I use. I notice it most when I'm doing time series. But here ...
4
votes
0answers
236 views

Equivalence of AIC and LOOCV under mismatched loss functions

Under certain conditions, AIC and LOOCV (leave-one-out cross validation) are asymptotically equivalent (Stone, 1977). Stone's paper is less than 4 pages long, but quite mathy, so I turn here for some ...
21
votes
1answer
23k views

How to compare models on the basis of AIC?

We have two models that use the same method to calculate log likelihood and the AIC for one is lower than the other. However, the one with the lower AIC is far more difficult to interpret. We are ...
48
votes
3answers
4k views

AIC,BIC,CIC,DIC,EIC,FIC,GIC,HIC,IIC — Can I use them interchangeably?

On p. 34 of his PRNN Brian Ripley comments that "The AIC was named by Akaike (1974) as 'An Information Criterion' although it seems commonly believed that the A stands for Akaike". Indeed, when ...
35
votes
3answers
17k views

Is it possible to calculate AIC and BIC for lasso regression models?

Is it possible to calculate AIC or BIC values for lasso regression models and other regularized models where parameters are only partially entering the equation. How does one determine the degrees of ...
43
votes
5answers
85k views

Negative values for AICc (corrected Akaike Information Criterion)

I have calculated AIC and AICc to compare two general linear mixed models; The AICs are positive with model 1 having a lower AIC than model 2. However, the values for AICc are both negative (model 1 ...
27
votes
3answers
3k views

AIC versus cross validation in time series: the small sample case

I am interested in model selection in a time series setting. For concreteness, suppose I want to select an ARMA model from a pool of ARMA models with different lag orders. The ultimate intent is ...
15
votes
5answers
13k views

Why applying model selection using AIC gives me non-significant p-values for the variables

I have some questions about the AIC and hope you can help me. I applied model selection (backward, or forward) based on the AIC on my data. And some of the selected variables ended up with a p-values >...
13
votes
2answers
2k views

Conflicting approaches to variable selection: AIC, p-values or both?

From what I understand, variable selection based on p-values (at least in regression context) is highly flawed. It appears variable selection based on AIC (or similar) is also considered flawed by ...
14
votes
0answers
4k views

What is the logic behind “rule of thumb” for meaningful differences in AIC?

I've been struggling to find meaningful guidelines for comparing models based on differences in AIC. I keep coming back to the rule of thumb offered by Burnham & Anderson 2004, pp. 270-272: ...
10
votes
2answers
5k views

What exactly is “stepwise model selection”?

Although the merits of stepwise model selection has been discussed previously, it is becoming unclear to me what exactly is "stepwise model selection" or "stepwise regression". I thought I understood ...
10
votes
2answers
2k views

Why information criterion (not adjusted $R^2$) are used to select appropriate lag order in time series model?

In time series models, like ARMA-GARCH, to select appropriate lag or order of the model different information criterion, like AIC, BIC, SIC etc, are used. My question is very simple, why donot we ...
6
votes
1answer
3k views

Linear mixed effects model and - multiplicity issue and adjusting for p-values

In our randomized controlled trial, we used linear mixed effects models to test differences between groups in changes from baseline to six months while adjusting for important covariates. We ran ...
4
votes
2answers
4k views

Comparing AIC among models with different amounts of data

I have a data set with many missing observations for certain parameters (NA values) in it. I have been performing model selection using AIC. Based on AIC scores I have reduce the model to the form <...
10
votes
2answers
11k views

What exactly is the Box-Jenkins method for ARIMA processes?

The Wikipedia page says that Box-Jenkins is a method of fitting an ARIMA model to a time series. Now, if I want to fit an ARIMA model to a time series, I will open up SAS, call ...
3
votes
3answers
7k views

Model comparison with AIC based on different sample size

Let's assume I have two models M1 and M2: M1: y ~ x1 + x2 + x3 M2: y ~ x1 + x2 + x3 + x4 Since variable x4 has some missing values the sample size of M2 is ...
3
votes
1answer
540 views

Can one give an example(s) of when non-nested AIC model comparison is not useful for model selection?

Note: The question here is not the same as this one. Indeed, as an answer to that question the answer below was closed as unrelated, together with the suggestion (credit @gung) to ask a separate ...
24
votes
5answers
4k views

What is the upside of treating a factor as random in a mixed model?

I have a problem embracing the benefits of labeling a model factor as random for a few reasons. To me it appears like in almost all cases the optimal solution is to treat all of the factors as fixed. ...
43
votes
1answer
111k views

Negative values for AIC in General Mixed Model [duplicate]

I'm trying to select the best model by the AIC in the General Mixed Model test. The best model is the model with the lowest AIC, but all my AIC's are negative! So is the biggest negative AIC the ...
11
votes
2answers
10k views

Comparing between random effects structures in a linear mixed-effects model

During a recently asked question about linear mixed-effects models I was told that one should not compare between models with different random effects structures using likelihood ratio tests. Up until ...
20
votes
2answers
5k views

Comparing AIC of a model and its log-transformed version

The essence of my question is this: Let $Y \in \mathbb{R}^n$ be a multivariate normal random variable with mean $\mu$ and covariance matrix $\Sigma$. Let $Z := \log(Y)$, i.e. $Z_i = \log(Y_i), i \...
18
votes
1answer
1k views

Does BIC try to find a true model?

This question is a follow-up or attempt to clear up possible confusion regarding a topic I and many others find a bit difficult, regarding the difference between AIC and BIC. In a very nice answer by @...
8
votes
1answer
9k views

Model selection: can I compare the AIC from models of count data between linear and poisson models?

I am modeling count data (with offset / exposure parameter). My modeling strategy is use of a Poisson model and a negative binomial regression model. I compare model AICs, which are about -760 for my ...
8
votes
4answers
16k views

When AIC and Adjusted $R^2$ lead to different conclusions

I hope it's okay to ask theoretically driven R questions here. R has given me the following results from my 'tournament of models'. All models are entirely distinct except from 3 basic control ...
15
votes
4answers
18k views

Number of parameters in an artificial neural network for AIC

How can I calculate the number of parameters in an artificial neural network in order to calculate its AIC?
6
votes
1answer
1k views

AIC and BIC criterion for Model selection, how is it used in this paper?

I'm reading Model selection and inference: Facts and fiction by Leeb & Pötscher (2005) (link), in this paper they look at an example in linear regression: Let $$Y_i = \alpha x_{i1}+\beta x_{i2}+\...
3
votes
2answers
504 views

A guide to regularization strategies in regression

I'm looking for some sort of guideline as when it is appropriate to use which forms of regularization and a comparison of the advantages / disadvantages of the various forms. So something that ...
4
votes
2answers
126 views

Optimality of AIC w.r.t. loss functions used for evaluation

Under certain conditions, AIC is an efficient model selection criterion. I understand this roughly as if AIC will tend to select the model that will yield the largest expected likelihood of a new data ...