Questions tagged [model-comparison]
Comparing two or more models fit to a common data set. It can be part of the process of "model selection".
563
questions
1
vote
1
answer
5
views
Right way of comparing my model's result with state of the arts
I know that I have to use the same datasets and experimental settings. In this way, I don't have to run their codes. What if I used different datasets? Is it OK to use their code from GitHub to get ...
- 21
0
votes
1
answer
38
views
Bayesian evidence with Sequential Monte Carlo and an unnormalized likelihood function: a contradiction?
There is a contradiction in my understanding of Sequential Monte Carlo for estimating Bayesian evidence for model comparison:
Marginal likelihood (aka normalizing constant, aka Bayesian evidence) ...
- 417
0
votes
0
answers
9
views
Calculate effect size Cohen's w (omega) for a chi-square test in model comparison
I have two logistic mixed-effects models, nested within each other and differing in only one fixed variable:
...
- 145
1
vote
0
answers
26
views
Comparing Sensitivities of Microscopic and Macroscopic Models (Travel Demand)
A simulation output for each microscopic travel demand model and two further macroscopic MDCEV models - transformed versions of the microscopic model - are given for a base scenario. Moreover, three ...
- 31
2
votes
1
answer
19
views
Regression model comparison query - AIC suggesting a non-significant model is better than an alternative significant model
Wondering if anyone can help. I’m trying to compare two regression models with one predictor to see which best describes the data.
Model one is a linear model (y = ax + b) with R2 = .036, F = 3.047, p ...
0
votes
0
answers
4
views
comparison of two different mediators for the models with the same DV and IVs
Based on my theoretical arguments, I have two competing mediators (M1 and M2), each of which possibly mediates the relationship between my independent (X) and dependent (Y) variable. I implemented two ...
- 1
0
votes
0
answers
13
views
Comparing 3D distribution
I want to compare observations with a set of simulated data, but I'm struggling to find the best way of doing so:
My observations consist on a set of datapoints (~100) in a 3 dimensional space
I ...
0
votes
0
answers
9
views
Pairwise comparison test for out-of-sample MSEs
In order to compare the out-of-sample forecasting accuracy of two competing models, I am trying to implement the equal accuracy test proposed in this article: http://www.timberlake-consultancy.com/...
0
votes
0
answers
8
views
comparing the performance of a neural network with the prediction of an analytical model: question on the test dataset
i have trained a small bayesian neural network by splitting a dataset in the usual training and test dataset.
Now i also have an analytical model whose parameters needs to be estimated by mcmc ...
- 261
1
vote
1
answer
34
views
Statistically compare R^2 values from linear regression models with same outcome but different predictors and Ns
I have two linear regression models predicting the same outcome with the same data, but the predictors are different and the sample sizes slightly vary due to missing values on some predictors but not ...
- 11
1
vote
0
answers
19
views
What is compared in a model comparison? [closed]
What is the purpose of model comparison? Is it to make the same predictions about the response variable, using the same regression formula of predictor variables, but with varying specifications (like ...
- 23
1
vote
0
answers
27
views
Can I use a paired t-test to assess whether there are statistically significant differences between the backtesting results of two forecasting models?
I developed a new forecasting model with the aim of replacing an older model. I conducted 12 backtests and need to predict only one quarter ahead. Overall, my new model performs better than the older ...
- 11
1
vote
0
answers
19
views
How can I compare one full PCA model to two smaller ones?
I have nearly 30 variables going in to a large PCA, but the variables really fall into two conceptual categories. I want to test whether leaving all the variables to correlate freely with one another ...
- 11
0
votes
0
answers
25
views
How to adjacent two non-overlapping short time series?
I am facing a situation where I am given a relatively short time series (in the worst-case scenario it may have up to 15 observations). During the time series we modify one external factor that we ...
0
votes
0
answers
17
views
How to compare the decline in results between multiple groups
I've tested a number of fruit samples for weight loss over time (shelf-life assessment).
I want to compare weight loss between the different samples.
My initial idea was to use a linear regression ...
- 103
1
vote
1
answer
45
views
How to find the appropriate model to apply (GAMLSS)/Approving statistical thinking
I need to compute percentile curves using the LMS method (from the GAMLSS package/models) with Age and Height as predictors.
What is the best way to determine which equation (with which transformation ...
- 79
0
votes
0
answers
59
views
Compare multinomial logistic regressions with multiple dependent variables
My dataset looks like this:
...
- 1
0
votes
0
answers
29
views
Do AIC improvements make multicollinearity 'worth it'? (and an analagous question about GAM concurvity and fREML)
To what extent is it the case that I can ignore multicollinearity if adding in those terms improves my AIC?
Frankly I'm a little rusty on my theory, but I've seen some people suggesting that AIC is a ...
Tyrone Sawyer
3
votes
1
answer
31
views
Model section when AIC and logLik differ
To analyze my ordinal (Liker scale ratings) I used the clmm function for the ordinal package. Since I have fixed and random ...
- 33
2
votes
2
answers
225
views
Normal or lognormal? After comparing the fit of those two models by calculating a likelihood ratio (LR), how to pool LRs from several data sets?
Using the method of section 6.7.2 of Burnham and Anderson, Model selection and multimodel inference: a practical information-theoretic approach, 2nd edition, one can compute the likelihood ratio ...
- 18.2k
0
votes
0
answers
13
views
Statistic tests for model comparison when both have high performance
It is correct to apply methods as McNemar test or 5x2CV paired t test in order to compare two classification models with high accuracy? For example, Logistic regression and KNN in the iris dataset. I'...
0
votes
0
answers
37
views
Transfer learning model comparisons
Is there a good test to highlight the top performing models in the following two scenarios? Is a t-test acceptable? The metrics used for assessing models are RMSE and relative RMSE (i.e., ...
- 313
0
votes
0
answers
16
views
Comparing nested simple and multiple regression models
I have a question regarding the intepretation of results from two regression models, these are based on the CAPM and APT (Arbitrage Price Theory)
The models are presented in the picture below:
Where E(...
0
votes
0
answers
58
views
Model comparison for regression: significant gain in rmse
I am comparing the predictive power of two regression models. RMSE gain is $13\%$ using a full model compared to reduced one on the test set. I do not want to compromise on model complexity hence ...
0
votes
0
answers
35
views
To the likelihood of what does the loglikelihood in AIC, BIC refer?
Both Akaike's information criterion and Bayesian information criterion are calculated from the loglikelihood, eg AIC= -2 log-likelihood +2k.
To the likelihood of what does it refer to? The likelihood ...
0
votes
0
answers
14
views
Comparing models using different training sets
I try to compare the forecasting performance of several models. I do it for two situations: normal and extreme cases. My dataset set is not big.
One of the models (gradient boosting) requiers a ...
- 111
0
votes
0
answers
36
views
Compare predictions of 10 models
I am aiming to compare predictions made using 10 different models with no reference to the observed value. The objective is two folds : i- check if the difference between the models predictions are ...
- 1
0
votes
0
answers
27
views
What measure to use for comparison of non-linear models with very small number of data points
My question is on the details and subtleties of comparing models in non-linear regression.
Situation: I want to find out which function fits a set of data points better. Therefore, I'm looking for ...
0
votes
0
answers
25
views
In which situation will clustering gives similar results as classification?
Suppose you have a labeled data set $(X,Y)$ where $Y$ is the target variable taking its value from $\{1,...,K \}$. Imagine you run a classification model on this dataset, which gives you good ...
- 13
3
votes
1
answer
196
views
How can I predict/interpret results of Poisson regression?
We are working on a project where we want to find the pure premium for which we first want to fit Poisson regression.
Data Used: data(dataCars) from insauranceData package in R
We have an offset term ...
- 31
0
votes
0
answers
58
views
Chi square for comparing image and models
In this question, the data is a 2D array which represents an image. Each element in the array is the flux or brightness of a given pixel.
As a second step, portions of the image are masked (pixel ...
- 111
0
votes
1
answer
19
views
Comparison problem: GAS (Generalized Autoregressive Score) and GARCH
My goal is to show that t-GAS is better than t-GARCH, so I am trying to analyze some data about Crude Oil volatility comparing these models with functions in R packages (...
- 1
0
votes
0
answers
19
views
How to choose ARMA order when comparing two time series?
I am comparing the return series of 2 ETFs. Both return series are stationary. I estimated the AIC information criterion for ARMA model with p and q from 0 to 5. I found the lowest AIC for ETF 1 with ...
- 145
0
votes
0
answers
46
views
How to compare two segmentation models using k-fold cross validation?
I have a small dataset of 87 images. I have done 5-fold cross-validation, training and validating two models on the same data splits. Thus, for each image I have an accuracy score (Dice score). It ...
- 1
1
vote
0
answers
47
views
what does it mean when the NLL becomes negative?
i was comparing the results of 3 different techniques for regression task( Deep ensembles, variational inference and concrete dropout) and i got these results
from the table looks like everything is ...
- 261
3
votes
0
answers
37
views
AIC comparison of linear models with Gaussian likelihood (equivalent to square loss) and Laplace likelihood (equivalent to absolute loss)
As is discussed in another question I posted, it is legitimate to compare the AIC values for two linear models using the same features but differing in their likelihoods, such as Gaussian and Laplace. ...
- 47.2k
2
votes
0
answers
41
views
Comparing GLM's with Different Response Distributions Using AIC [duplicate]
Let's say I fit the following two models in R:
...
- 321
0
votes
0
answers
14
views
Methods for evaluating out-of-sample performance on a single dataset with multiple stages of model comparison
Question
Suppose you want to evaluate the out-of-sample performance of $M$ unknown models with data from an $N$ (independent) observation dataset. However, $M$ are only known after comparing the out-...
- 711
1
vote
1
answer
34
views
Which metric to use to evaluate and compare meta models?
I'm developing meta models based on Zero Inflated Model (sklego.meta.ZeroInflatedRegressor)
That is, they are models that have two tasks to perform:
Train a ...
- 31
0
votes
1
answer
35
views
Model comparison between inherently different machine-learning algorithms
For example, how to fairly compare the performance of XGBoost/LightGBM to the performance of a Feed-Forward Neural Network? Especially regarding (1) Data Preprocessing and (2) Hyperparameter Tuning? ...
- 148
1
vote
1
answer
38
views
n value in the calculation of BIC
This question is building on this post, and on the general formula for BIC.
BIC = kln(n) - 2ln(L)
I need a bit of help understanding what the term number of ...
- 75
1
vote
1
answer
56
views
Comparing $R^2$ between Gaussian GLM with identity link and with $\log$ link
It is known that $R^2$ should not be compared between two regressions where one uses features $X_1,\dots ,X_n$ to predict $Y$ and the other uses those same features to predict $\log(Y)$.
However, this ...
- 47.2k
0
votes
0
answers
27
views
How to statistically determine if two files have identical content?
I'm writing a duplicate file finder application, and I do not want to have to read every byte of every file to determine which files are duplicates.
Hard drive storage is segmented into clusters of X ...
- 101
1
vote
1
answer
196
views
How to compare variance explained in two regression models?
Let's say I have a measurement of height of 200 individuals as $y_1$ and fit a simple linear regression model as:
$y_1 = x_1 + x_2 + \epsilon_1$
where $x_1$ and $x_2$ are some subject metadata
I ...
- 231
0
votes
0
answers
7
views
Choosing between univariate GLMMs to assess inclusion in a multivariate model
I am currently doing a project on environmental determinants of malaria vector distributions. I'm using remote sensing data for environmental variables linked via GIS. I have run univariate (binomial) ...
0
votes
0
answers
13
views
How would you compare competing Bayesian hierarchical models that are estimating the same time-series?
Suppose we have two models $\mathcal{M}_1$ and $\mathcal{M}_2$ that attempt to estimate a time-series $\boldsymbol{\theta}$. Let's assume that I actually know the true values for $\boldsymbol{\theta}$....
- 1,777
4
votes
2
answers
280
views
Compare GLM AICs with different likelihoods?
If I have a generalized linear model (GLM) with a particular likelihood, and I have another GLM of the same data (say nested within the first model), I can compare the model performance using Akaike ...
- 47.2k
4
votes
2
answers
431
views
Valid model comparison/selection? Poisson, negative binomial, zero-inflated poisson, z-i negative binomial with AIC
I want to model counts of an event in a pre-post design. A sample dataset could look like this:
...
- 61
0
votes
1
answer
15
views
Which factor should I consider to choose the best model, stationarity or lowest MSE?
Recently, I did a project where I modelled a dataset of Gold price. I used ARIMA(1,1,1) and ARIMA(2,2,3) to model the data. The results that I get was ARIMA(1,1,1) was not stationary but the MSE value ...
0
votes
1
answer
82
views
