27
votes
Is it possible that the AIC and BIC give totally different model selections?
It is possible indeed. As explained at https://methodology.psu.edu/AIC-vs-BIC, "BIC penalizes model complexity more heavily. The only way they should disagree is when AIC chooses a larger model than ...
- 20.9k
21
votes
Why isn't Akaike information criterion used more in machine learning?
AIC and BIC are used, e.g. in stepwise regression. They are actually part of a larger class of "heuristics", which are also used. For example the DIC (Deviance Information Criterion) is often used in ...
- 2,671
18
votes
Is it possible to calculate AIC and BIC for lasso regression models?
I was struggling a lot with a way how to calculate AIC and BIC for glmnet models. However, after quite a lot of searching, I found on the third page of google results the answer. It can be found here. ...
- 301
18
votes
Accepted
Is it possible that AIC = BIC?
As a reminder:
$$AIC = - 2 \log \mathcal{L}(\hat{\theta}|X)+2k $$
$$BIC = - 2 \log \mathcal{L}(\hat{\theta}|X)+k \ln(n)$$
So for what values of $n$ is $2 = \ln(n)$?
- 1,056
17
votes
Accepted
In what applications do we prefer Model Selection over Model Averaging?
Model specification is a better approach than either model selection (by which most people refer really to feature selection) or model averaging. Here are some pros and cons.
model specification ...
- 98.5k
15
votes
Does BIC try to find a true model?
The Information Criterion by Schwarz (1978) was designed with the feature that it asymptotically chooses the model with the higher posterior odds, i.e. the model with the higher likelihood given the ...
- 1,632
13
votes
Using BIC to estimate the number of k in KMEANS
This is basically eyaler's solution, with a few notes.. I just typed it out if someone wanted a quick copy/paste:
Notes:
eyalers 4th comment is incorrect np.log is already a natural log, no change ...
13
votes
Is it possible that the AIC and BIC give totally different model selections?
Short answer: yes, it is very possible. The two apply different penalties based on the number of estimated parameters (2k for AIC vs ln(n) x k for BIC, where k is the number of estimated parameters ...
- 549
13
votes
In what applications do we prefer Model Selection over Model Averaging?
Single models are much easier to interpret than ensembles. They are therefore often more easily accepted by non-technical users, and are easier to troubleshoot (e.g., when a prediction is way off).
...
- 131k
12
votes
Accepted
Mclust model selection
Solution found:
So, to restate the question, why does the Mclust function default to the model with the highest BIC value as the "best" model?
Great question! ...
- 2,400
12
votes
If the AIC and the BIC are asymptotically equivalent to cross validation, is it possible to dispense with a test set when using them?
AIC is asymptotically equivalent to leave-1-out cross-validation (LOOCV) (Stone 1977) and BIC is equivalent to leave-k-out cross-validation (LKOCV) where $k=n[1−1/(\log(n)−1)]$, with $n=$ sample size (...
- 3,417
11
votes
Negative BIC in k-means
This may result useful for someone else.
I was puzzled with the mclust package because I tried Gaussian mixture models to check whether my data followed a uni- or multi-modal Gaussian distribution.
I ...
- 327
11
votes
AIC, BIC and log likelihood which more important?
If you compare two models one of which is "bigger" than the other (i.e. has all the parameters of the other one and some more), the loglikelihood will always be larger for the bigger model, ...
- 28.3k
10
votes
Accepted
Use BIC or AIC as approximation for Bayesian Model Averaging
I don't understand the first sentence very well, but if your asking whether you should use BIC or AIC to approximate BMA than the answer is fairly straightforward.
You would use BIC rather than AIC ...
- 4,042
10
votes
Zero-inflated Poisson regression Vuong test: Raw, AIC- or BIC-corrected results
I am convinced that it is incorrect to use the Vuong test -- in any of its forms -- as a test for zero-inflation. I have had a paper "The misuse of the Vuong test for non-nested models to test for ...
- 166
10
votes
On George Box, Galit Shmueli and the scientific method?
Let me start with the pithy quote by George Box, that "all models are wrong, but some are useful". This statement is an encapsulation of the methodological approach of "positivism", which is a ...
- 133k
10
votes
How to interpret negative values for -2LL, AIC, and BIC?
The bottom line is that (as Jeremy Miles says) the value of the negative log-likelihood doesn't really matter, only differences between the negative log-likelihoods. But you might still wonder why you ...
- 47.4k
10
votes
Does information criteria (AIC, BIC and DIC...) imply "causality"?
There is nothing inherently causal about any score. A score encodes assumptions about the underlying model. If the assumptions are met, a score can yield a causal model.
Score-based causal discovery ...
- 2,439
8
votes
AIC and BIC criterion for Model selection, how is it used in this paper?
In my answer here I show that in a case like the present one, in which we test nested models against each other, the minimum AIC rule selects the larger model (i.e., rejects the null) if the ...
- 34.8k
8
votes
Accepted
How do you write the AIC and BIC of a regression model in terms of the coefficient-of-determination?
In Gaussian regression, the MLE for the coefficient vector is equivalent to the OLS estimator, and the MLE for the error variance has its usual form relating to the residual sum-of-squares in the ...
- 133k
8
votes
Accepted
Why would a smaller AIC than BIC lead to an increased chance of both overfitting and underfitting?
Both Akaike information criterion and Bayesian information criterion balance the number of parameters and the magnitude of the likelihood:
$$
AIC=2k-2\log(\hat{L}),\\
BIC=k\log (n)-2\log(\hat{L})
$$
...
- 4,497
7
votes
Why is the bayesian information criterion called that way?
BIC, sometimes called the Schwarz information criterion (SIC) was introduced by Gideon Schwartz in 1975. Here is that paper. It's not very long.
Both AIC and BIC address the model evaluation problem ...
- 211
7
votes
Accepted
Compare linear regression slopes between non-nested models with differing dataset sizes
Welcome here.
A simple solution is to define a new binary variables which takes the value $1$ for the subset of the data and $0$ for all other observations not included in the subset. Then you can ...
- 3,661
6
votes
Comparison between MDL and BIC
The Bayesian Infomration Criterion (BIC) is given as:
\begin{equation}\label{eq_BIC_FINAL}
BIC = \log f\left( {\bf{x}}|\hat{{\bf{\theta}}}_i ; H_i\right) - \frac{1}{2} \log \left| I\left(\hat{{\bf{\...
- 169
6
votes
Accepted
Variable selection vs Model selection
Sometimes modelers separate variable selection into a distinct step in model development. For instance, they would first perform exploratory analysis, research the academic literature and industry ...
- 62.3k
6
votes
Is there any reason to prefer the AIC or BIC over the other?
Very briefly:
AIC approximately minimizes the prediction error (in terms of expected Kullback-Leibler (KL) divergence, which measures the difference between the true model and the estimated model) ...
- 3,417
6
votes
Which model should be chosen using AICc/BIC and P-value?
The AIC, the BIC and the $p$-values all address different questions. For feature selection (variable selection, model selection), only the former two are relevant. See e.g. Rob J. Hyndman's blog posts ...
- 69.5k
6
votes
BIC vs. intuition
It's not an obvious question at all! In fact, I think there may be some disagreement even among statisticians.
My view is that you should never let the computer do your thinking for you. Don't ...
- 128k
6
votes
If the AIC and the BIC are asymptotically equivalent to cross validation, is it possible to dispense with a test set when using them?
Not really an answer to your question, but a bit long for a comment. Asymptotic equivalence boils down to saying: if the sample is very large and the number of parameters not that large, how you ...
- 8,718
6
votes
Accepted
AIC/BIC for a segmented regression model?
In principle yes. If you know the segments (i.e., if they are given exogenously), then the segmented regression model just corresponds to a certain kind of interaction model and information criteria ...
- 16.1k
Only top scored, non community-wiki answers of a minimum length are eligible