Questions tagged [information-criteria]
Information-criteria for model selection such as aic, bic, dic, mdl and others
39
questions
0
votes
0
answers
17
views
VAR models: Constant inclusion and information criteria
My question is straightforward: can information criterion be used to determine whether to include a constant in a VAR model?
My imagined workflow is as follows: since most packages include a ...
- 460
0
votes
0
answers
69
views
Justification for using information criteria (AIC, AICc, BIC) in ARIMA time series model selection
Intro
As far as I understand, AIC, AICc, and BIC information criteria can be used when different models $M_j$ that differ in the number of estimated parameters $K_j$, which are applied to the same $\...
- 301
1
vote
0
answers
52
views
In search of parsimony...Can/Should Information Criterion be used as Cost Functions in the Hyperparameter Tuning of Regularization Models?
When tuning regularized models, two techniques appear to be especially popular at the moment:
Cross-validation performance on train & validation splits (the third, test/holdout set is not used in ...
- 11
7
votes
2
answers
244
views
Selecting ARIMA orders by ACF/PACF vs. by information criteria
We keep on getting questions here about selecting ARIMA model orders based on ACF/PACF plots. This is the older methodology proposed by Box and Jenkins.
More modern tools like the ...
- 111k
1
vote
1
answer
50
views
n value in the calculation of BIC
This question is building on this post, and on the general formula for BIC.
BIC = kln(n) - 2ln(L)
I need a bit of help understanding what the term number of ...
- 75
1
vote
0
answers
21
views
AIC for different variables
I have been wondering, can AIC be safely used for model selection when the datasets contain a different number of variables? What I mean is, can a model predicting house price that contains "...
- 11
2
votes
1
answer
116
views
Tree models and information criterion
I am currently trying to compare the complexity of models. Among the models I have are some trees. The trees are not parametric models, hence they don't have the notion of 'trainable parameters' that ...
- 1,506
0
votes
0
answers
35
views
Difference between Bayesian Information Criteria and Approximate Bayesian Computation as model selection
My question is not very technical and more like a discussion but I will be happy to have a technical input for the comparison b/w BIC and ABC.
I am trying to understand and use the best model ...
- 141
6
votes
3
answers
2k
views
Model Selection: AIC/BIC and Cross-Validation gives different conclusion
In general, there vast number of ways to select model/feature in machine learning or statistics. For example, empirical method like Cross-Validation, Bootstrap methods or in sample penalty such as AIC,...
- 532
4
votes
1
answer
87
views
Model selection criteria that represent a compromise between AIC and BIC
I am very familiar with the ideas and formula of the two popular model selection criteria AIC/AICc and BIC. When I use them for practical problems in chemometrics, the use of AIC/AICc often gives ...
- 43
1
vote
1
answer
331
views
Information criteria to select best Prophet model
How do you optimize your hyperparameters when using prophet for forecasting? I have been using cross validation and I don't know why no information criteria (such as AIC or BIC) has been implemented ...
- 19
5
votes
0
answers
101
views
Brief characterizations of AIC and BIC: how helpful are they?
I have found the following one-sentence characterizations of AIC and BIC in a lecture note:
AIC estimates the degree to which the predictive accuracy of the model will generalize to new data.
BIC ...
- 62.2k
3
votes
1
answer
51
views
Model selection: comparing Bayesian models with likelihood vs likelihood-free (Approximate Bayesian Computation)
I have two families of models that can possibly explain the data at hand.
One family is rather process-based, using fairly complicated simulations and Approximate Bayesian Computation to estimate the ...
- 131
1
vote
1
answer
499
views
Can AIC be used to select the best model with multiply imputed data (MICE)?
I have used the mice() package in R to impute some missing values and create a pooled linear regression model. I have also created another version but this time ...
Jameshbj9
0
votes
1
answer
95
views
Goodness of fit test for predictive model with small sample size
I am new to building the prediction model and would like to hear from you regarding the goodness of fit test (model calibration). Between Akaike information criteria (AIC), Hosmer-Lemeshow Test, ...
- 331
1
vote
0
answers
66
views
Minimum Description Length (MDL) criterion for polynomial regression
For data $ (y_{1},x_{1}),...(y_{n},x_{n})$, consider the polynomial regression $y_{i} = g(x_{i}) + \epsilon_{i} $ where $\epsilon_{i}$ are iid gaussian errors $N(0,\sigma^{2})$, and $g(x) = a_{0} + a_{...
- 11
0
votes
0
answers
596
views
Augmented Dickey Fuller test, determining the number of lags used
In a paper by Mark P. Taylor he conducts ADF tests on exchange rates and he describes the determination of lag number with "The number of lagged dependents that we need to include to induce ...
- 635
2
votes
1
answer
104
views
Information Criteria and Sample size
In the estimation of the univariate time series model, we need to determine the correct order. For the general ARMA (p,q) model, we can determine the true order by information criteria
$$AIC(p,q) = ln(...
- 890
1
vote
0
answers
31
views
Nonparametric estimation and kolmogorov sufficient statistics
Recently, I decided to revisit Cover and Thomas, and yesterday I encountered a very interesting passage in the chapter on Kolmogorov complexity:
What does this "different procedure" look ...
- 624
2
votes
0
answers
305
views
Forecasting: AIC, AICc and BIC VS Cross Validation for model selection (cases for different horizons)
The majority of the automatic model selection algorithms like auto.arima and ets (https://robjhyndman.com/publications/automatic-...
- 21
1
vote
1
answer
368
views
Does AIC criterion for variable selection in the least square method require the data to be normally distributed?
I'm studying Linear Regression from the book A Modern Approach to Regression with R by Simon J. Sheather (the 2009 edition). The chapter 7 (page 228), in which different criterion for variable ...
- 143
2
votes
1
answer
211
views
Is there any rationale for rules of thumb for maxlag selection?
I understand that the optimum number of lags can be selected using some information criterion (e.g. Akaike (AIC), Schwartz Bayesian Information Criterion (SBIC) etc.). However, in order to select the ...
- 850
1
vote
1
answer
531
views
How to get log-likelihood from squared deviance in Scikit Learn
The score() function computes D^2, the percentage of deviance explained, but I'd like to get the log-likelihood to calculate BIC. What's the formula to go from deviance to log-likelihood?
Score ...
- 113
1
vote
0
answers
155
views
More efficient way to find the model with the lowest AIC?
I am working on the zero-inflated Poisson model with a dataset, and trying to find a combination of variables that yields the lowest AIC. I used a series of for loops in R, but it takes forever to get ...
- 43
5
votes
1
answer
162
views
Model comparison with intractable likelihood using approximate Bayesian Computation
I have some models based on stochastic differential equations (SDEs). Because of the definition of these models, I can simulate data, but I cannot compute the likelihood function / distribution ...
- 2,501
1
vote
0
answers
406
views
The similarity between Mallows Cp and AIC?
It is possible to compute the log-likelihood used for AIC as $n /log(RSS/n) + const$ or even as $RSS/\sigma^2 + n\log(\sigma) + const$ considering the least-square or MLE scenario for linear and non-...
- 33
2
votes
0
answers
38
views
How can be information criteria used with non-independent samples?
I'm puzzled by the following. Many, if not all, of the Information Criteria (AIC, WAIC, LOOIC, PSIS, ...) rely on the independence of the samples -- that we can remove a part, that one point is ...
- 141
5
votes
1
answer
129
views
The Ambiguity in Schwarz Information Criterion Definition
Suppose there are 100 countries, $i = 1, 2, ..., 100$.
Let $b_i$ be the median birth weight of all new born boys in country #i in 2019.
Let $g_i$ be the median birth weight of all new born girls in ...
- 471
1
vote
1
answer
288
views
Differences between formulas for AIC and BIC
I have a question regarding the information criteria AIC and BIC:
I found different formulas for the AIC/BIC, the common ones including the likelihood $\mathcal{L}$ are
$$AIC = 2K - 2 ln(\mathcal{L})\...
- 63
1
vote
1
answer
1k
views
Log-Likelihood Computation for AIC & BIC
Considering $n$ observations that an be modelled by a Gaussian error model and two nested motion models with $p = 4$ and $p = 7$ parameters, I want to compute the log likelihoods $L$ given the Maximum ...
- 33
0
votes
1
answer
647
views
Criterion to Pick Number of hidden layers for MLP based on score
I'm training a classifier with a set of training data and checking the result with a set of test data. I'm using sklearn score based on all data and iteratively ...
- 111
3
votes
1
answer
162
views
Comparing model performance: do fixed and random effect regression models give different model ranking?
I am a graduate student in animal science.
I am comparing linear models that fit covariates of var1 and var2. These two covariates are decomposed from one quantity say F (inbreeding level of animal).
...
0
votes
0
answers
338
views
How can I compare the performance of model selected by AIC and K-fold cross validation?
guys. I try to compare the model selection criterions. The model is built as a linear regression with 20 explanatory variables. More precisely, what I did so far, is spilting the data into training(70%...
- 1
1
vote
2
answers
3k
views
Feature subset selection by stepwise regression for a random forest model?
I would like to build a random forest model for regression. I have an abundance of potential features, and I expect only some of them to have a significant impact on the target variable. In addition, ...
- 11
1
vote
1
answer
315
views
Clarification on Akaike's IC (AIC) and BIC for Expectation Maximization with time-changing parameters
I apologize in advance for the trivial question, but I need a clarification on the following issue. Suppose I have a generic model in state-space form described as
$$x_{t+1}=\phi_{t} x_{t}+w_{t+1}$$
$...
- 1,458
3
votes
0
answers
362
views
WAIC for model comparisons--overly conservative?
I'm having a hard time wrapping my head around the relationship between model posterior predictions and model comparisons via WAIC. Specifically, how do I interpret findings where a model including ...
- 31
6
votes
2
answers
996
views
Why is the bayesian information criterion called that way?
The word "Bayes" suggests that we are updating a distribution using data, to get a posterior distribution.
The fact that the Bayesian information criterion (BIC) is used to select a model from a set ...
- 2,459
0
votes
0
answers
65
views
is there a measure for the roughness of a contour plot
There has to be a measure for the difference between "instantaneous" change of "energy" along a line in a space compared to averaged changed of energy along a line.
I could take a smooth surface in ...
- 8,561
2
votes
0
answers
179
views
Using minimum description length for a categorical distribution
My data is as follows: in a routing node (check figure ), I can see the entry and exit gate of each packet, so I have pairs like this:
(1,5) (2,5) (1,6) ...
where ...
- 397