Skip to main content

Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

Filter by
Sorted by
Tagged with
34 votes
3 answers
24k views

Prerequisites for AIC model comparison

What are exactly the prerequisites, that need to be fulfilled for AIC model comparison to work? I just came around this question when I did comparison like this: ...
Tomas's user avatar
  • 6,193
4 votes
2 answers
5k views

AIC/BIC and data transformation

Can you use AIC/BIC to compare models on untransformed data with models on transformed data (such as log, inverse hyperbolic sine, etc.)? I.e. if a model using logged data gives an AIC = 53.62 and a ...
David King's user avatar
3 votes
3 answers
270 views

Comparing models with main effects and interactions

I have two models: Model 1: Only contains independent variable $x$, while $x$ is non-significant. Model 2: Contains $x$, $m$, and $x * m$, and $x * m$ is significant. How could I illustrate this ...
david's user avatar
  • 31
5 votes
4 answers
11k views

Model comparison with AIC based on different sample size

Let's assume I have two models M1 and M2: M1: y ~ x1 + x2 + x3 M2: y ~ x1 + x2 + x3 + x4 Since variable x4 has some missing values the sample size of M2 is ...
giordano's user avatar
  • 1,019
5 votes
1 answer
923 views

Can one give an example(s) of when non-nested AIC model comparison is not useful for model selection?

Note: The question here is not the same as this one. Indeed, as an answer to that question the answer below was closed as unrelated, together with the suggestion (credit @gung) to ask a separate ...
Carl's user avatar
  • 13.2k
13 votes
1 answer
7k views

Comparison negative binomial model and quasi-Poisson

I have run negative binomial and quasi-Poisson models based on an hypothesis testing approach. My final models using both methods have different covariates and interactions. It seems that there are no ...
Elena Spark's user avatar
6 votes
2 answers
5k views

Comparing AIC among models with different amounts of data

I have a data set with many missing observations for certain parameters (NA values) in it. I have been performing model selection using AIC. Based on AIC scores I have reduce the model to the form <...
colin's user avatar
  • 1,232
5 votes
2 answers
1k views

Compare GLM AICs with different likelihoods?

If I have a generalized linear model (GLM) with a particular likelihood, and I have another GLM of the same data (say nested within the first model), I can compare the model performance using Akaike ...
Dave's user avatar
  • 63.5k
2 votes
1 answer
2k views

Permutation test for model comparison?

I have two nested, non-linear models for the same data, and I want to test whether the more complex model explains significantly more variance. Due to a necessary smoothing step, my data aren't ...
Ruben van Bergen's user avatar
7 votes
3 answers
765 views

How is the relationship between two variables $X$ and $Y$ supposed to "explain" $R^2\text%$ of the variation of the data?

Suppose we have a linear regression and we calculate $R^2 = 0.81$. What do we mean when we say "the relationship between two variables $X$ and $Y$ explains $81\text%$ of the variation of the data&...
mathgeek's user avatar
  • 551
2 votes
1 answer
910 views

Tests of Forecast Accuracy for Nested Models

Can anyone explain why "classic" tests of forecast accuracy, (i.e. Diebold-Mariano test, Meese-Rogoff test and Morgan-Granger-Newbold test) are not suited for nested models? I could not find a good ...
mlieb's user avatar
  • 21
0 votes
0 answers
574 views

Comparing the fit of quasi-Poisson and negative binomial models

Is there any way to compare the fit of quasi-Poission and negative binomial models? If so, can it be done in R?
JKO's user avatar
  • 785
16 votes
3 answers
8k views

Similarity of two discrete fourier tranforms?

In climate modelling, you're looking for a models that can adequately portray the Earth's climate. This includes showing patterns that are semi-cyclical: things like the El Nino Southern Oscillation. ...
naught101's user avatar
  • 5,461
15 votes
1 answer
53k views

Difference between "in-sample" and "pseudo out-of-sample" forecasts

Is there an explicit difference between in-sample forecasts and pseudo out-of-sample forecasts. Both is meant in the context of evaluating and comparing forecasting models.
altabq's user avatar
  • 705
14 votes
2 answers
4k views

Is it wrong to compare multiple models on the same test set and choose the best model?

Suppose we split a dataset into 3 parts (train, validation, and test). I know that it's important to make sure the test set doesn't influence our decisions during model selection or hyperparameter ...
Abdulwahab Almestekawy's user avatar
13 votes
2 answers
7k views

Number of parameters in Markov model

I want to use BIC for HMM model selection: BIC = -2*logLike + num_of_params * log(num_of_data) So how do I count the number of parameters in the HMM model. ...
Sergey's user avatar
  • 606
6 votes
2 answers
1k views

How to do external validation of regression models

Very basic question here, so bear with me... I have a data set with 241 patients with 16 variables plus diagnosis (malignant vs benign). There are 3 previously published logistic regression formulas ...
K Owen - Reinstate Monica's user avatar
6 votes
3 answers
7k views

Can you compare p-values of Kolmogorov Smirnov tests of normality of two variables to say which is more normal?

I have applied the one sample Kolmogorov Smirnov test of normality to two variables and one has a larger p value but both are greater than .05. e.g., $x_1$ (p-value) = 0.09 $x_2$ (p-value) = 0.06 ...
blossom emerald's user avatar
6 votes
2 answers
4k views

Why may results from model with interaction term and stratified model be different?

Suppose I wanted to explore the relationship between smoking (X; yes/no) and an disease outcome (Y; eg. visual analogue scale of depression from 0 to 10). But, I know that irrespective of X, Y is ...
bobmcpop's user avatar
  • 1,303
5 votes
1 answer
520 views

Mixed-effect model single term deletion -- should I change my random effects?

In short I recently had a little conversation on the lme4 project's GitHub on how to properly test the significance of effects in a mixed-effect model, which made ...
Arthur Spoon's user avatar
5 votes
0 answers
5k views

AIC or ANOVA to compare models?

What are the relative merits of each approach, and which circumstances call for one rather than the other? To some extent I have a specific example in mind, which I've discussed here. In that example ...
user1205901 - Слава Україні's user avatar
5 votes
1 answer
816 views

Why do you need non-linear regression if you can use a linear one to fit any kind of curvature to your data?

Polynomial regression fits a non-linear model to the data. But as a statistical estimation problem it's still linear in the sense that the regression function $h\left(\Theta, X\right)$ is linear in ...
mathgeek's user avatar
  • 551
4 votes
1 answer
6k views

Interpretation of (scale of) AIC, AICc and BIC when comparing different models

I'm trying to fit a model to a time series, but I am pretty confused as to which is the best. I'm looking at an arima model, and ets model and an stlf model, which each performed best within their ...
SiKiHe's user avatar
  • 465
2 votes
1 answer
331 views

summary of lm to compare models instead of ANOVA

What does it mean to use anova to compare models? For example, if I have two models $$ \text{model}_1 \colon y = b_0 + b_1 x_1 + b_2 x_2 $$ and $$ \text{...
stats _student's user avatar
2 votes
2 answers
865 views

AIC calculation with very low negative log likelihood

I am using AIC formula (AIC=2k−2lnL) to compare different exponential models. I know that this formula is used to penalize complexed models (with high number of parameters). The problem I have is that ...
user265113's user avatar
1 vote
1 answer
2k views

what is a good measure of goodness of fit for survival models that can be used for comparison between models?

I don't see any of such measure in the output from coxph() in R (Cox proportional Hazard model). Is there a goodness of fit measure for survival models similar to R2 for linear regression? Update: ...
hehe's user avatar
  • 733
1 vote
0 answers
14 views

Evaluate Value at Risk (VaR) models with different VaR backtesting approaches - averaging p-values?

I estimated the Value at Risk of a time series of log returns with different approaches and models. Now I want to compare the models and choose the model that most accurately estimated the Value at ...
Isabel's user avatar
  • 31
1 vote
1 answer
105 views

How to find the appropriate model to apply (GAMLSS)/Approving statistical thinking

I need to compute percentile curves using the LMS method (from the GAMLSS package/models) with Age and Height as predictors. What is the best way to determine which equation (with which transformation ...
stats's user avatar
  • 89
1 vote
1 answer
170 views

Comparing $R^2$ between Gaussian GLM with identity link and with $\log$ link

It is known that $R^2$ should not be compared between two regressions where one uses features $X_1,\dots ,X_n$ to predict $Y$ and the other uses those same features to predict $\log(Y)$. However, this ...
Dave's user avatar
  • 63.5k
0 votes
0 answers
17 views

Choosing one method for modeling Value at Risk (VaR) over another - determining the "best" model

I estimated the Value at Risk of a time series of log returns with different approaches and models. Now I want to compare the models and chose the model that most accurately estimated the Value at ...
Isabel's user avatar
  • 31
15 votes
2 answers
80k views

Comparing two linear regression models

I would like to compare two linear regression models which represent degradation rates of a mRNA over time under two different conditions. The data for each model collected independently. Here is ...
Rooz's user avatar
  • 151
14 votes
5 answers
6k views

When to use multiple models for prediction?

This is a fairly general question: I have typically found that using multiple different models outperforms one model when trying to predict a time series out of sample. Are there any good papers ...
Shane's user avatar
  • 12.5k
14 votes
4 answers
2k views

What is the relationship between ANOVA to compare means of several groups and ANOVA to compare nested models?

I've so far seen ANOVA used in two ways: First, in my introductory statistics text, ANOVA was introduced as a way to compare means of three or more groups, as an improvement over pairwise comparison, ...
Austin's user avatar
  • 753
12 votes
3 answers
7k views

Comparing regression models on count data

I recently fit 4 multiple regression models for the same predictor/response data. Two of the models I fit with Poisson regression. ...
Daniel Standage's user avatar
11 votes
3 answers
16k views

Model comparison between an ARIMA model and a regression model

I'm really having trouble finding out how to compare ARIMA and regression models. I understand how to evaluate ARIMA models against each other, and different types of regression models (ie: ...
Brett's user avatar
  • 111
9 votes
3 answers
8k views

Is overfitted model with higher AUC on test sample better than not overfitted one

i am participating in a challange in which I have created a model that performs 70% AUC on train set and 70% AUC on hold-out test set. The other participant has created a model that performs 96% AUC ...
MiksL's user avatar
  • 177
8 votes
1 answer
213 views

How can I compare my model to a technically invalid model?

I've created nice little nonlinear model relating survival probability to length in salmon. I fit it assuming binomial errors and minimizing the negative log likelihood. I've been asked to compare it ...
Gregor Thomas's user avatar
7 votes
2 answers
3k views

Encompassing Tests to compare models in R

I recently read a paper in which an academic compared two competing models of voting theory. One of the advanced diagnostic tests he did was to try and see whether either of the models could "formally ...
HenryBukowski's user avatar
7 votes
1 answer
7k views

Residual plot for nonlinear regression

I have a couple of questions regarding performance of nonlinear regression models. Are the residuals from a nonlinear regression model supposed to be randomly distributed too (as in linear ...
Learner's user avatar
  • 85
7 votes
0 answers
638 views

F-test to determine whether more than two sets of data differ

Here is the context for my question: I understand that you can fit the same model to two different datasets separately and then fit the model to the datasets pooled together as a way to discern ...
Angela's user avatar
  • 540
7 votes
1 answer
2k views

Different results using Brier score and Logarithmic scoring rule

Let $X_i\sim B(\pi_i), \text{for }i=1,2,\cdots,n$. I have two models and I want to compare which of them forecast better. Model 1: Estimates the parameters with maximum likelihood. Model 2: ...
F.F.'s user avatar
  • 549
6 votes
1 answer
16k views

How to interpret and compare models in Cox regression?

I am trying to interpret the results of a Cox regression; I am doing a PhD in medicine. I love statistics but my question is still pretty basic, I think, and I did not find an answer in previous ...
torwart's user avatar
  • 305
6 votes
2 answers
758 views

Metrics for comparing estimated lists to a 'true' list

I'm wondering what the best ways to compare (possibly ranked) lists when we know what the true ranking is and also the variable that decides the ranking. Say this is the top 10 of a certain list, we ...
dcl's user avatar
  • 2,762
5 votes
1 answer
2k views

How to test significance in shift of a variable taken an other variable into the model (suest) in R?

I want to test in R whether an additional variable $x_2$ taken into a model influences an other variable significantly. (Note: This a revised repost of one in Stack Overflow where there was no ...
jay.sf's user avatar
  • 802
5 votes
1 answer
198 views

Bayesian formulation of best subset regression

We know Ridge is equivalent to using a Gaussian prior and Lasso is equivalent to using a double exponential prior. What is the Bayesian interpretation (implied prior) for the best subset regression? ...
user avatar
5 votes
3 answers
690 views

If summarizing stats from multiple models is it meaningful to report a mean AIC?

I am currently summarizing results from several groups of models. Is it meaningful to report a mean AIC for each group of models? If not then how best to give a summary measure for each model ...
Sideshow Bob's user avatar
  • 1,485
5 votes
1 answer
1k views

Alternatives for Diebold-Mariano test when comparing the best forecast among many against a benchmark

Suppose I encounter a new forecasting method and I wish to evaluate it against a benchmark. I can obtain forecasts from the two methods and compare them to actual realizations and thus obtain the ...
Richard Hardy's user avatar
5 votes
1 answer
1k views

Mean absolute percentage error with respect to predictions

A friend of mine has suggested that instead of using mean absolute percentage error, i.e. $$ \frac{1}{N}\sum_{i=0}^N \left| \frac{A_i - P_i}{A_i} \right| $$ (where $A_i$ denotes an actual value, $P_i$ ...
ignoring_gravity's user avatar
5 votes
2 answers
384 views

Permutation test comparing nested non-linear models with an exchangeable dummy variable

This question is closely related to an earlier question, but I realized my case was actually a lot more specific than the way I formulated it there, in ways that I think merit a separate answer. I ...
Ruben van Bergen's user avatar
4 votes
1 answer
390 views

R lmer model: add factors or reduce factors

In mixed effect models, do you add factors one by one? Or do you reduce the factors one by one? What I am doing is as follows. Are there any problems with the steps? Build a full model: ...
SuperDuperMario's user avatar