# Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

540 questions
Filter by
Sorted by
Tagged with
12 views

### Bayesian Model selection vs Model comparison

Is anyone aware of any articles or book chapters about the distinction between model selection and model comparison in bayesian multilevel modeling? I am fitting several competing growth models using ...
• 542
23 views

### How to statistically determine if two files have identical content?

I'm writing a duplicate file finder application, and I do not want to have to read every byte of every file to determine which files are duplicates. Hard drive storage is segmented into clusters of X ...
• 101
1 vote
29 views

### How to compare variance explained in two regression models?

Let's say I have a measurement of height of 200 individuals as $y_1$ and fit a simple linear regression model as: $y_1 = x_1 + x_2 + \epsilon_1$ where $x_1$ and $x_2$ are some subject metadata I ...
• 231
6 views

### Choosing between univariate GLMMs to assess inclusion in a multivariate model

I am currently doing a project on environmental determinants of malaria vector distributions. I'm using remote sensing data for environmental variables linked via GIS. I have run univariate (binomial) ...
11 views

### How would you compare competing Bayesian hierarchical models that are estimating the same time-series?

Suppose we have two models $\mathcal{M}_1$ and $\mathcal{M}_2$ that attempt to estimate a time-series $\boldsymbol{\theta}$. Let's assume that I actually know the true values for $\boldsymbol{\theta}$....
• 1,615
100 views

### Compare GLM AICs with different likelihoods?

If I have a generalized linear model (GLM) with a particular likelihood, and I have another GLM of the same data (say nested within the first model), I can compare the model performance using Akaike ...
• 35.7k
262 views

### Valid model comparison/selection? Poisson, negative binomial, zero-inflated poisson, z-i negative binomial with AIC

I want to model counts of an event in a pre-post design. A sample dataset could look like this: ...
9 views

### Which factor should I consider to choose the best model, stationarity or lowest MSE?

Recently, I did a project where I modelled a dataset of Gold price. I used ARIMA(1,1,1) and ARIMA(2,2,3) to model the data. The results that I get was ARIMA(1,1,1) was not stationary but the MSE value ...
58 views

### $R^2$ model comparison in test data

I want to compare two models of the form: ...
1 vote
28 views

### Independent Samples T-test vs Paired Samples T-test for hypothesis testing (machine learning algorithms comparison)

The scenario is as follows: I have 1 dataset and I want to compare the performance of different algorithms. Let's say A, B, C. I train each algorithm a 1000 times in one training loop. At the ...
• 11
1 vote
26 views

### Specification of regression model based on two data sources

Suppose that I have two different datasets from surveys in which the dependent variable is measured on different scales and I want to evaluate the effect the survey method (i.e. the measuring scale) ...
• 11
1 vote
4 views

### Rank Correlation for Paired Predictor to compate 2 GLM models: appropiate objects and p-value

Dear Professor Harrell and users, Thank you for this post: How to assess the incremental additive information of a new predictor in a Cox model using R?, it has been really helpful for my work. I am ...
60 views

### Is there a metric to compare machine learning models which takes into account how many trainable parameters they have?

Assume we have two models for the same task. One with 1K parameters and one with 1M and they achieve accuracy as follows Model Params Accuracy small 1 000 0.8 big 1 000 000 0.9 I would like to ...
71 views

### SEM model comparison

I have problem choosing a better measurement model between two (i.e., A and B). Both models have acceptable fit indices. For example, the RMSEA of A is .056; the RMSEA of B is .067. By RMSEA, A seems ...
1 vote
138 views

1 vote
19 views

### Comparing models with partially observed data and limited supervision

Setup: Suppose we have $n$ questions and for each question $m$ answers. For each question, a model, Model A, selects one answer out of $m$. Note that for each question, zero, one, or more answers ...
• 670
25 views

### Model selection: comparing Bayesian models with likelihood vs likelihood-free (Approximate Bayesian Computation)

I have two families of models that can possibly explain the data at hand. One family is rather process-based, using fairly complicated simulations and Approximate Bayesian Computation to estimate the ...
• 121
1 vote
60 views

### GARCH model sensitivity to distribution assumptions

I am trying to fit an ARMA(4,4)- GARCH(1,1) model to return data, where the distribution of returns is highly leptokurtic. I plan to see whether autocorrelations exist in the data even after ...
23 views

### Comparing model fit on data aggregated at different levels

I'm fitting a non-linear model to data aggregated at 5 different levels and want to deduce at which level the model fits the data best, i.e. compare the model fits at the 5 different aggregation ...
22 views

### Is it okay to use AIC values generated from summary() for glmer?

I am conducting a random effect logistic regression using glmer in R. I have 13 different predictor variables, which I am evaluating at 4 different spatial scales (i.e. I have 13 variables derived ...
82 views

### Why does a Decision Tree algorithm outperform the Random Forest Algorithm in certain cases?

Currently I write my master thesis that deals with the binary prediction of university dropouts (dropout - yes/no). In the thesis, I compare the performance of three different classification ...
21 views

### Time Series Forecasting with Different Time Horizon for Comparing Models?

At the moment I am dealing with a time series problem. The data I have is about 6 years and in daily frequency. I want to try out different models on the data and I came up with an experiment: ...
20 views

### Combination of multiple datasets to compare models with experiments

I work on metal fatigue lifetime estimation of material under different loading conditions. I have performed a series of experiments on a certain material to determine the lifetime of the material for ...
28 views

### what is the difference between RMSE and Mc error (in Bayesian estimation)?

Please, I want to know if there is a difference between Root mean square error (RMSE) and Monte carlo error (Mc error). I need to compare my models but in Bayesian modelling I found a value like Mc ...
1 vote
23 views

### Is there a consensus on bad practice when doing model comparison in deep learning?

Lately I've been thinking about how to maintain the Equity property when Benchmarking DL for research. On Wikipedia, Equity property is defined as "All systems should be fairly compared." ...
• 143
31 views

### Null model comparisons with mixed models

I have a question about the general practice of null model comparisons. I understand why they can be useful in assessing model quality for prediction. My question comes when involving varying (random) ...
• 51
1 vote