# Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

563 questions
Filter by
Sorted by
Tagged with
1 vote
5 views

### Right way of comparing my model's result with state of the arts

I know that I have to use the same datasets and experimental settings. In this way, I don't have to run their codes. What if I used different datasets? Is it OK to use their code from GitHub to get ...
• 21
38 views

### Bayesian evidence with Sequential Monte Carlo and an unnormalized likelihood function: a contradiction?

There is a contradiction in my understanding of Sequential Monte Carlo for estimating Bayesian evidence for model comparison: Marginal likelihood (aka normalizing constant, aka Bayesian evidence) ...
• 417
9 views

### Calculate effect size Cohen's w (omega) for a chi-square test in model comparison

I have two logistic mixed-effects models, nested within each other and differing in only one fixed variable: ...
• 145
1 vote
26 views

### Comparing Sensitivities of Microscopic and Macroscopic Models (Travel Demand)

A simulation output for each microscopic travel demand model and two further macroscopic MDCEV models - transformed versions of the microscopic model - are given for a base scenario. Moreover, three ...
• 31
19 views

### Regression model comparison query - AIC suggesting a non-significant model is better than an alternative significant model

Wondering if anyone can help. I’m trying to compare two regression models with one predictor to see which best describes the data. Model one is a linear model (y = ax + b) with R2 = .036, F = 3.047, p ...
4 views

### comparison of two different mediators for the models with the same DV and IVs

Based on my theoretical arguments, I have two competing mediators (M1 and M2), each of which possibly mediates the relationship between my independent (X) and dependent (Y) variable. I implemented two ...
• 1
13 views

### Comparing 3D distribution

I want to compare observations with a set of simulated data, but I'm struggling to find the best way of doing so: My observations consist on a set of datapoints (~100) in a 3 dimensional space I ...
9 views

### Pairwise comparison test for out-of-sample MSEs

In order to compare the out-of-sample forecasting accuracy of two competing models, I am trying to implement the equal accuracy test proposed in this article: http://www.timberlake-consultancy.com/...
8 views

### comparing the performance of a neural network with the prediction of an analytical model: question on the test dataset

i have trained a small bayesian neural network by splitting a dataset in the usual training and test dataset. Now i also have an analytical model whose parameters needs to be estimated by mcmc ...
• 261
1 vote
34 views

### Statistically compare R^2 values from linear regression models with same outcome but different predictors and Ns

I have two linear regression models predicting the same outcome with the same data, but the predictors are different and the sample sizes slightly vary due to missing values on some predictors but not ...
• 11
1 vote
19 views

### What is compared in a model comparison? [closed]

What is the purpose of model comparison? Is it to make the same predictions about the response variable, using the same regression formula of predictor variables, but with varying specifications (like ...
1 vote
27 views

### Can I use a paired t-test to assess whether there are statistically significant differences between the backtesting results of two forecasting models?

I developed a new forecasting model with the aim of replacing an older model. I conducted 12 backtests and need to predict only one quarter ahead. Overall, my new model performs better than the older ...
1 vote
19 views

### How can I compare one full PCA model to two smaller ones?

I have nearly 30 variables going in to a large PCA, but the variables really fall into two conceptual categories. I want to test whether leaving all the variables to correlate freely with one another ...
25 views

### How to adjacent two non-overlapping short time series?

I am facing a situation where I am given a relatively short time series (in the worst-case scenario it may have up to 15 observations). During the time series we modify one external factor that we ...
17 views

### How to compare the decline in results between multiple groups

I've tested a number of fruit samples for weight loss over time (shelf-life assessment). I want to compare weight loss between the different samples. My initial idea was to use a linear regression ...
1 vote
45 views

### How to find the appropriate model to apply (GAMLSS)/Approving statistical thinking

I need to compute percentile curves using the LMS method (from the GAMLSS package/models) with Age and Height as predictors. What is the best way to determine which equation (with which transformation ...
• 79
59 views

### Compare multinomial logistic regressions with multiple dependent variables

My dataset looks like this: ...
29 views

### Do AIC improvements make multicollinearity 'worth it'? (and an analagous question about GAM concurvity and fREML)

To what extent is it the case that I can ignore multicollinearity if adding in those terms improves my AIC? Frankly I'm a little rusty on my theory, but I've seen some people suggesting that AIC is a ... 31 views

### Model section when AIC and logLik differ

To analyze my ordinal (Liker scale ratings) I used the clmm function for the ordinal package. Since I have fixed and random ...
• 33
225 views

### Normal or lognormal? After comparing the fit of those two models by calculating a likelihood ratio (LR), how to pool LRs from several data sets?

Using the method of section 6.7.2 of Burnham and Anderson, Model selection and multimodel inference: a practical information-theoretic approach, 2nd edition, one can compute the likelihood ratio ...
• 18.2k
13 views

### Statistic tests for model comparison when both have high performance

It is correct to apply methods as McNemar test or 5x2CV paired t test in order to compare two classification models with high accuracy? For example, Logistic regression and KNN in the iris dataset. I'...
37 views

### Transfer learning model comparisons

Is there a good test to highlight the top performing models in the following two scenarios? Is a t-test acceptable? The metrics used for assessing models are RMSE and relative RMSE (i.e., ...
• 313
16 views

### Comparing nested simple and multiple regression models

I have a question regarding the intepretation of results from two regression models, these are based on the CAPM and APT (Arbitrage Price Theory) The models are presented in the picture below: Where E(...
58 views

### Model comparison for regression: significant gain in rmse

I am comparing the predictive power of two regression models. RMSE gain is $13\%$ using a full model compared to reduced one on the test set. I do not want to compromise on model complexity hence ...
35 views

### To the likelihood of what does the loglikelihood in AIC, BIC refer?

Both Akaike's information criterion and Bayesian information criterion are calculated from the loglikelihood, eg AIC= -2 log-likelihood +2k. To the likelihood of what does it refer to? The likelihood ...
14 views

### Comparing models using different training sets

I try to compare the forecasting performance of several models. I do it for two situations: normal and extreme cases. My dataset set is not big. One of the models (gradient boosting) requiers a ...
• 111
36 views

### Compare predictions of 10 models

I am aiming to compare predictions made using 10 different models with no reference to the observed value. The objective is two folds : i- check if the difference between the models predictions are ...
27 views

### What measure to use for comparison of non-linear models with very small number of data points

My question is on the details and subtleties of comparing models in non-linear regression. Situation: I want to find out which function fits a set of data points better. Therefore, I'm looking for ...
25 views

### In which situation will clustering gives similar results as classification?

Suppose you have a labeled data set $(X,Y)$ where $Y$ is the target variable taking its value from $\{1,...,K \}$. Imagine you run a classification model on this dataset, which gives you good ...
196 views

### How can I predict/interpret results of Poisson regression?

We are working on a project where we want to find the pure premium for which we first want to fit Poisson regression. Data Used: data(dataCars) from insauranceData package in R We have an offset term ...
58 views

### Chi square for comparing image and models

In this question, the data is a 2D array which represents an image. Each element in the array is the flux or brightness of a given pixel. As a second step, portions of the image are masked (pixel ...
• 111
19 views

### Comparison problem: GAS (Generalized Autoregressive Score) and GARCH

My goal is to show that t-GAS is better than t-GARCH, so I am trying to analyze some data about Crude Oil volatility comparing these models with functions in R packages (...
19 views

### How to choose ARMA order when comparing two time series?

I am comparing the return series of 2 ETFs. Both return series are stationary. I estimated the AIC information criterion for ARMA model with p and q from 0 to 5. I found the lowest AIC for ETF 1 with ...
• 145
46 views

### How to compare two segmentation models using k-fold cross validation?

I have a small dataset of 87 images. I have done 5-fold cross-validation, training and validating two models on the same data splits. Thus, for each image I have an accuracy score (Dice score). It ...
1 vote
47 views

### what does it mean when the NLL becomes negative?

i was comparing the results of 3 different techniques for regression task( Deep ensembles, variational inference and concrete dropout) and i got these results from the table looks like everything is ...
• 261
37 views

### AIC comparison of linear models with Gaussian likelihood (equivalent to square loss) and Laplace likelihood (equivalent to absolute loss)

As is discussed in another question I posted, it is legitimate to compare the AIC values for two linear models using the same features but differing in their likelihoods, such as Gaussian and Laplace. ...
• 47.2k
41 views

### Comparing GLM's with Different Response Distributions Using AIC [duplicate]

Let's say I fit the following two models in R: ...
• 321
14 views

### Methods for evaluating out-of-sample performance on a single dataset with multiple stages of model comparison

Question Suppose you want to evaluate the out-of-sample performance of $M$ unknown models with data from an $N$ (independent) observation dataset. However, $M$ are only known after comparing the out-...
1 vote
34 views

### Which metric to use to evaluate and compare meta models?

I'm developing meta models based on Zero Inflated Model (sklego.meta.ZeroInflatedRegressor) That is, they are models that have two tasks to perform: Train a ...
35 views

### Model comparison between inherently different machine-learning algorithms

For example, how to fairly compare the performance of XGBoost/LightGBM to the performance of a Feed-Forward Neural Network? Especially regarding (1) Data Preprocessing and (2) Hyperparameter Tuning? ...
1 vote
38 views

### n value in the calculation of BIC

This question is building on this post, and on the general formula for BIC. BIC = kln(n) - 2ln(L) I need a bit of help understanding what the term number of ...
1 vote
56 views

### Comparing $R^2$ between Gaussian GLM with identity link and with $\log$ link

It is known that $R^2$ should not be compared between two regressions where one uses features $X_1,\dots ,X_n$ to predict $Y$ and the other uses those same features to predict $\log(Y)$. However, this ...
• 47.2k
27 views

### How to statistically determine if two files have identical content?

I'm writing a duplicate file finder application, and I do not want to have to read every byte of every file to determine which files are duplicates. Hard drive storage is segmented into clusters of X ...
• 101
1 vote
196 views

### How to compare variance explained in two regression models?

Let's say I have a measurement of height of 200 individuals as $y_1$ and fit a simple linear regression model as: $y_1 = x_1 + x_2 + \epsilon_1$ where $x_1$ and $x_2$ are some subject metadata I ...
• 231
7 views

### Choosing between univariate GLMMs to assess inclusion in a multivariate model

I am currently doing a project on environmental determinants of malaria vector distributions. I'm using remote sensing data for environmental variables linked via GIS. I have run univariate (binomial) ...
13 views

### How would you compare competing Bayesian hierarchical models that are estimating the same time-series?

Suppose we have two models $\mathcal{M}_1$ and $\mathcal{M}_2$ that attempt to estimate a time-series $\boldsymbol{\theta}$. Let's assume that I actually know the true values for $\boldsymbol{\theta}$....
• 1,777
280 views

### Compare GLM AICs with different likelihoods?

If I have a generalized linear model (GLM) with a particular likelihood, and I have another GLM of the same data (say nested within the first model), I can compare the model performance using Akaike ...
• 47.2k
431 views

### Valid model comparison/selection? Poisson, negative binomial, zero-inflated poisson, z-i negative binomial with AIC

I want to model counts of an event in a pre-post design. A sample dataset could look like this: ...