# Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

80 questions
Filter by
Sorted by
Tagged with
24k views

### Prerequisites for AIC model comparison

What are exactly the prerequisites, that need to be fulfilled for AIC model comparison to work? I just came around this question when I did comparison like this: ...
• 6,193
5k views

### AIC/BIC and data transformation

Can you use AIC/BIC to compare models on untransformed data with models on transformed data (such as log, inverse hyperbolic sine, etc.)? I.e. if a model using logged data gives an AIC = 53.62 and a ...
270 views

### Comparing models with main effects and interactions

I have two models: Model 1: Only contains independent variable $x$, while $x$ is non-significant. Model 2: Contains $x$, $m$, and $x * m$, and $x * m$ is significant. How could I illustrate this ...
• 31
11k views

### Model comparison with AIC based on different sample size

Let's assume I have two models M1 and M2: M1: y ~ x1 + x2 + x3 M2: y ~ x1 + x2 + x3 + x4 Since variable x4 has some missing values the sample size of M2 is ...
• 1,019
923 views

### Can one give an example(s) of when non-nested AIC model comparison is not useful for model selection?

Note: The question here is not the same as this one. Indeed, as an answer to that question the answer below was closed as unrelated, together with the suggestion (credit @gung) to ask a separate ...
• 13.2k
7k views

### Comparison negative binomial model and quasi-Poisson

I have run negative binomial and quasi-Poisson models based on an hypothesis testing approach. My final models using both methods have different covariates and interactions. It seems that there are no ...
• 405
5k views

### Comparing AIC among models with different amounts of data

I have a data set with many missing observations for certain parameters (NA values) in it. I have been performing model selection using AIC. Based on AIC scores I have reduce the model to the form <...
• 1,232
1k views

### Compare GLM AICs with different likelihoods?

If I have a generalized linear model (GLM) with a particular likelihood, and I have another GLM of the same data (say nested within the first model), I can compare the model performance using Akaike ...
• 63.5k
2k views

### Permutation test for model comparison?

I have two nested, non-linear models for the same data, and I want to test whether the more complex model explains significantly more variance. Due to a necessary smoothing step, my data aren't ...
• 7,559
765 views

### How is the relationship between two variables $X$ and $Y$ supposed to "explain" $R^2\text%$ of the variation of the data?

Suppose we have a linear regression and we calculate $R^2 = 0.81$. What do we mean when we say "the relationship between two variables $X$ and $Y$ explains $81\text%$ of the variation of the data&...
• 551
910 views

### Tests of Forecast Accuracy for Nested Models

Can anyone explain why "classic" tests of forecast accuracy, (i.e. Diebold-Mariano test, Meese-Rogoff test and Morgan-Granger-Newbold test) are not suited for nested models? I could not find a good ...
• 21
574 views

### Comparing the fit of quasi-Poisson and negative binomial models

Is there any way to compare the fit of quasi-Poission and negative binomial models? If so, can it be done in R?
• 785
8k views

### Similarity of two discrete fourier tranforms?

In climate modelling, you're looking for a models that can adequately portray the Earth's climate. This includes showing patterns that are semi-cyclical: things like the El Nino Southern Oscillation. ...
• 5,461
53k views

### Difference between "in-sample" and "pseudo out-of-sample" forecasts

Is there an explicit difference between in-sample forecasts and pseudo out-of-sample forecasts. Both is meant in the context of evaluating and comparing forecasting models.
• 705
4k views

### Is it wrong to compare multiple models on the same test set and choose the best model?

Suppose we split a dataset into 3 parts (train, validation, and test). I know that it's important to make sure the test set doesn't influence our decisions during model selection or hyperparameter ...
7k views

### Number of parameters in Markov model

I want to use BIC for HMM model selection: BIC = -2*logLike + num_of_params * log(num_of_data) So how do I count the number of parameters in the HMM model. ...
• 606
1k views

### How to do external validation of regression models

Very basic question here, so bear with me... I have a data set with 241 patients with 16 variables plus diagnosis (malignant vs benign). There are 3 previously published logistic regression formulas ...
7k views

### Can you compare p-values of Kolmogorov Smirnov tests of normality of two variables to say which is more normal?

I have applied the one sample Kolmogorov Smirnov test of normality to two variables and one has a larger p value but both are greater than .05. e.g., $x_1$ (p-value) = 0.09 $x_2$ (p-value) = 0.06 ...
4k views

### Why may results from model with interaction term and stratified model be different?

Suppose I wanted to explore the relationship between smoking (X; yes/no) and an disease outcome (Y; eg. visual analogue scale of depression from 0 to 10). But, I know that irrespective of X, Y is ...
• 1,303
520 views

### Mixed-effect model single term deletion -- should I change my random effects?

In short I recently had a little conversation on the lme4 project's GitHub on how to properly test the significance of effects in a mixed-effect model, which made ...
5k views

### AIC or ANOVA to compare models?

What are the relative merits of each approach, and which circumstances call for one rather than the other? To some extent I have a specific example in mind, which I've discussed here. In that example ...
816 views

### Why do you need non-linear regression if you can use a linear one to fit any kind of curvature to your data?

Polynomial regression fits a non-linear model to the data. But as a statistical estimation problem it's still linear in the sense that the regression function $h\left(\Theta, X\right)$ is linear in ...
• 551
6k views

### Interpretation of (scale of) AIC, AICc and BIC when comparing different models

I'm trying to fit a model to a time series, but I am pretty confused as to which is the best. I'm looking at an arima model, and ets model and an stlf model, which each performed best within their ...
• 465
331 views