Skip to main content

Questions tagged [information-criteria]

Information-criteria for model selection such as aic, bic, dic, mdl and others

Filter by
Sorted by
Tagged with
0 votes
0 answers
16 views

Better Prediction model with large DIC value

I am using Integrated Nested Laplace Approximation (INLA) to predict birds population.In my (INLA) models, I've noticed that when I include auto spatial correlations, my predictions are good and make ...
Usman YousafZai's user avatar
1 vote
2 answers
96 views

Why do information criteria provide consistent estimation of the lag order?

Consider the following $AR(p)$ model $$Y_{t}=\sum_{j=1}^{p}\phi_{j}Y_{t-j}+\epsilon_{t},$$ where $\epsilon_{t}\sim i.i.d\ \mathcal{N}(0,\sigma^{2}).$ Say we have samples $t=1,\dots, T$. The true lag ...
JacobsonRadical's user avatar
0 votes
1 answer
23 views

How many lags to insert into a GARCH(m,p) model?

My question might be trivial, but the doubt arises due to different ways of dealing with modeling that I have found in different research papers. In particular, I was able to observe that (in time ...
Giuseppe Vonella's user avatar
2 votes
1 answer
457 views

How to understand the information entropy calculation in entropy weight method?

I found the input of information entropy calculation in the entropy weight method is not probabilities. I can not understand why it works. In most entropy weight method introductions, the information ...
Alex's user avatar
  • 93
2 votes
1 answer
270 views

How to compare WAIC value when they are negative?

How to compare WAIC values when they are negative? Is it still the lower the better? I got two complex Gaussian models with continuous probability density and the WAIC value are -8351 and -7321, ...
weizhang's user avatar
1 vote
0 answers
54 views

In search of parsimony...Can/Should Information Criterion be used as Cost Functions in the Hyperparameter Tuning of Regularization Models?

When tuning regularized models, two techniques appear to be especially popular at the moment: Cross-validation performance on train & validation splits (the third, test/holdout set is not used in ...
FiddleBat's user avatar
10 votes
2 answers
760 views

Selecting ARIMA orders by ACF/PACF vs. by information criteria

We keep on getting questions here about selecting ARIMA model orders based on ACF/PACF plots. This is the older methodology proposed by Box and Jenkins. More modern tools like the ...
Stephan Kolassa's user avatar
2 votes
1 answer
85 views

n value in the calculation of BIC

This question is building on this post, and on the general formula for BIC. BIC = kln(n) - 2ln(L) I need a bit of help understanding what the term number of ...
Joseph K.'s user avatar
1 vote
0 answers
26 views

AIC for different variables

I have been wondering, can AIC be safely used for model selection when the datasets contain a different number of variables? What I mean is, can a model predicting house price that contains "...
coder1122's user avatar
3 votes
1 answer
358 views

Tree models and information criterion

I am currently trying to compare the complexity of models. Among the models I have are some trees. The trees are not parametric models, hence they don't have the notion of 'trainable parameters' that ...
Lucas Morin's user avatar
  • 1,665
0 votes
0 answers
51 views

Difference between Bayesian Information Criteria and Approximate Bayesian Computation as model selection

My question is not very technical and more like a discussion but I will be happy to have a technical input for the comparison b/w BIC and ABC. I am trying to understand and use the best model ...
Usman YousafZai's user avatar
9 votes
3 answers
4k views

Model Selection: AIC/BIC and Cross-Validation gives different conclusion

In general, there vast number of ways to select model/feature in machine learning or statistics. For example, empirical method like Cross-Validation, Bootstrap methods or in sample penalty such as AIC,...
Bayesian's user avatar
  • 572
4 votes
1 answer
215 views

Model selection criteria that represent a compromise between AIC and BIC

I am very familiar with the ideas and formula of the two popular model selection criteria AIC/AICc and BIC. When I use them for practical problems in chemometrics, the use of AIC/AICc often gives ...
sshwang's user avatar
  • 43
1 vote
1 answer
672 views

Information criteria to select best Prophet model

How do you optimize your hyperparameters when using prophet for forecasting? I have been using cross validation and I don't know why no information criteria (such as AIC or BIC) has been implemented ...
N-Light's user avatar
  • 19
5 votes
0 answers
122 views

Brief characterizations of AIC and BIC: how helpful are they?

I have found the following one-sentence characterizations of AIC and BIC in a lecture note: AIC estimates the degree to which the predictive accuracy of the model will generalize to new data. BIC ...
Richard Hardy's user avatar
3 votes
1 answer
93 views

Model selection: comparing Bayesian models with likelihood vs likelihood-free (Approximate Bayesian Computation)

I have two families of models that can possibly explain the data at hand. One family is rather process-based, using fairly complicated simulations and Approximate Bayesian Computation to estimate the ...
WaterFox's user avatar
  • 131
2 votes
1 answer
963 views

Can AIC be used to select the best model with multiply imputed data (MICE)?

I have used the mice() package in R to impute some missing values and create a pooled linear regression model. I have also created another version but this time ...
user avatar
0 votes
1 answer
162 views

Goodness of fit test for predictive model with small sample size

I am new to building the prediction model and would like to hear from you regarding the goodness of fit test (model calibration). Between Akaike information criteria (AIC), Hosmer-Lemeshow Test, ...
R Beginner's user avatar
1 vote
0 answers
79 views

Minimum Description Length (MDL) criterion for polynomial regression

For data $ (y_{1},x_{1}),...(y_{n},x_{n})$, consider the polynomial regression $y_{i} = g(x_{i}) + \epsilon_{i} $ where $\epsilon_{i}$ are iid gaussian errors $N(0,\sigma^{2})$, and $g(x) = a_{0} + a_{...
Anthony's user avatar
  • 11
0 votes
0 answers
822 views

Augmented Dickey Fuller test, determining the number of lags used

In a paper by Mark P. Taylor he conducts ADF tests on exchange rates and he describes the determination of lag number with "The number of lagged dependents that we need to include to induce ...
J3lackkyy's user avatar
  • 735
2 votes
1 answer
133 views

Information Criteria and Sample size

In the estimation of the univariate time series model, we need to determine the correct order. For the general ARMA (p,q) model, we can determine the true order by information criteria $$AIC(p,q) = ln(...
1190's user avatar
  • 1,142
1 vote
0 answers
32 views

Nonparametric estimation and kolmogorov sufficient statistics

Recently, I decided to revisit Cover and Thomas, and yesterday I encountered a very interesting passage in the chapter on Kolmogorov complexity: What does this "different procedure" look ...
user3716267's user avatar
3 votes
0 answers
470 views

Forecasting: AIC, AICc and BIC VS Cross Validation for model selection (cases for different horizons)

The majority of the automatic model selection algorithms like auto.arima and ets (https://robjhyndman.com/publications/automatic-...
leonidas's user avatar
1 vote
1 answer
681 views

Does AIC criterion for variable selection in the least square method require the data to be normally distributed?

I'm studying Linear Regression from the book A Modern Approach to Regression with R by Simon J. Sheather (the 2009 edition). The chapter 7 (page 228), in which different criterion for variable ...
Minh Khôi's user avatar
2 votes
1 answer
419 views

Is there any rationale for rules of thumb for maxlag selection?

I understand that the optimum number of lags can be selected using some information criterion (e.g. Akaike (AIC), Schwartz Bayesian Information Criterion (SBIC) etc.). However, in order to select the ...
1muflon1's user avatar
  • 905
1 vote
1 answer
661 views

How to get log-likelihood from squared deviance in Scikit Learn

The score() function computes D^2, the percentage of deviance explained, but I'd like to get the log-likelihood to calculate BIC. What's the formula to go from deviance to log-likelihood? Score ...
Alex F's user avatar
  • 113
1 vote
0 answers
232 views

More efficient way to find the model with the lowest AIC?

I am working on the zero-inflated Poisson model with a dataset, and trying to find a combination of variables that yields the lowest AIC. I used a series of for loops in R, but it takes forever to get ...
Shin's user avatar
  • 43
6 votes
1 answer
194 views

Model comparison with intractable likelihood using approximate Bayesian Computation

I have some models based on stochastic differential equations (SDEs). Because of the definition of these models, I can simulate data, but I cannot compute the likelihood function / distribution ...
LiKao's user avatar
  • 2,671
1 vote
0 answers
561 views

The similarity between Mallows Cp and AIC?

It is possible to compute the log-likelihood used for AIC as $n /log(RSS/n) + const$ or even as $RSS/\sigma^2 + n\log(\sigma) + const$ considering the least-square or MLE scenario for linear and non-...
Miau's user avatar
  • 133
2 votes
0 answers
42 views

How can be information criteria used with non-independent samples?

I'm puzzled by the following. Many, if not all, of the Information Criteria (AIC, WAIC, LOOIC, PSIS, ...) rely on the independence of the samples -- that we can remove a part, that one point is ...
My Work's user avatar
  • 141
5 votes
1 answer
135 views

The Ambiguity in Schwarz Information Criterion Definition

Suppose there are 100 countries, $i = 1, 2, ..., 100$. Let $b_i$ be the median birth weight of all new born boys in country #i in 2019. Let $g_i$ be the median birth weight of all new born girls in ...
Shang Zhang's user avatar
1 vote
1 answer
369 views

Differences between formulas for AIC and BIC

I have a question regarding the information criteria AIC and BIC: I found different formulas for the AIC/BIC, the common ones including the likelihood $\mathcal{L}$ are $$AIC = 2K - 2 ln(\mathcal{L})\...
Feuerraeder's user avatar
1 vote
1 answer
2k views

Log-Likelihood Computation for AIC & BIC

Considering $n$ observations that an be modelled by a Gaussian error model and two nested motion models with $p = 4$ and $p = 7$ parameters, I want to compute the log likelihoods $L$ given the Maximum ...
Miau's user avatar
  • 133
0 votes
1 answer
982 views

Criterion to Pick Number of hidden layers for MLP based on score

I'm training a classifier with a set of training data and checking the result with a set of test data. I'm using sklearn score based on all data and iteratively ...
Mefitico's user avatar
  • 111
3 votes
1 answer
188 views

Comparing model performance: do fixed and random effect regression models give different model ranking?

I am a graduate student in animal science. I am comparing linear models that fit covariates of var1 and var2. These two covariates are decomposed from one quantity say F (inbreeding level of animal). ...
Pattrapoljk Sumreddee's user avatar
0 votes
0 answers
386 views

How can I compare the performance of model selected by AIC and K-fold cross validation?

guys. I try to compare the model selection criterions. The model is built as a linear regression with 20 explanatory variables. More precisely, what I did so far, is spilting the data into training(70%...
Chaos's user avatar
  • 1
1 vote
2 answers
3k views

Feature subset selection by stepwise regression for a random forest model?

I would like to build a random forest model for regression. I have an abundance of potential features, and I expect only some of them to have a significant impact on the target variable. In addition, ...
GenH's user avatar
  • 11
1 vote
1 answer
377 views

Clarification on Akaike's IC (AIC) and BIC for Expectation Maximization with time-changing parameters

I apologize in advance for the trivial question, but I need a clarification on the following issue. Suppose I have a generic model in state-space form described as $$x_{t+1}=\phi_{t} x_{t}+w_{t+1}$$ $...
Fr1's user avatar
  • 1,478
3 votes
0 answers
417 views

WAIC for model comparisons--overly conservative?

I'm having a hard time wrapping my head around the relationship between model posterior predictions and model comparisons via WAIC. Specifically, how do I interpret findings where a model including ...
E.M.'s user avatar
  • 31
6 votes
2 answers
1k views

Why is the bayesian information criterion called that way?

The word "Bayes" suggests that we are updating a distribution using data, to get a posterior distribution. The fact that the Bayesian information criterion (BIC) is used to select a model from a set ...
user56834's user avatar
  • 2,877
0 votes
0 answers
70 views

is there a measure for the roughness of a contour plot

There has to be a measure for the difference between "instantaneous" change of "energy" along a line in a space compared to averaged changed of energy along a line. I could take a smooth surface in ...
EngrStudent's user avatar
  • 9,580
2 votes
0 answers
184 views

Using minimum description length for a categorical distribution

My data is as follows: in a routing node (check figure ), I can see the entry and exit gate of each packet, so I have pairs like this: (1,5) (2,5) (1,6) ... where ...
dsign's user avatar
  • 397