# Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

450 questions
Filter by
Sorted by
Tagged with
20 views

### The ratio of two Bayes factors for two opposite one-tailed hypotheses

I am trying to understand how Bayesian inference works, so this might be a very simple question. I have an experiment where I test two hypotheses predicting opposite results. Let’s say, hypothesis 1 (...
21 views

### Model Comparison with Regression

For a project I have to test a mediation model. I will do this with regression in R. The steps I take are the regular steps: Test if the IV has a relation with the DV (Regression) Test if the IV has ...
27 views

### random effect with one observation per group improves AIC drastically — explain

Now that I've posted an example (for a different purpose) at How will random effects with only 1 observation affect a generalized linear mixed model? (resulting in creating an account here, and being ...
20 views

### Checking for consistency and selecting the best among multiple measurement techniques

I am tasked with comparing $N$ statistical analysis methods (where $N$ is relatively small, say $N=5$), possibly not independent, for estimating some quantities $X,Y,Z$ based on some given data. Each ...
11 views

### Model Selestion - Compare AIC of data and its subset

I am trying to use AIC to assess the difference between replicates. In my experiment, each treatment has three replicates. The data from each replicates was fitted exponential model. And then combine ...
13 views

### Ranking of models on dev set does not match ranking on test

Using my train/dev/test split, I optimized four different model's hyper parameters over the development set. Suppose the ranking based on F1 score on the development set was: model 1 model 2 model 3 ...
18 views

### Can I use a (paired) t-test to compare the accuracies of classification models on a hold out test set?

When comparing the Brier scores or logloss/cross-entropy for multiple models on a hold-out test set, how can I be sure the 'best' model doesn't just have the lowest loss due to chance? When can I be ...
18 views

### Using model comparison to deal with collinearity in linear mixed models

I am working on a set given to me, which involves fitting logistic models with a couple of predictors, some of which are nearly perfectly correlated (.9), imagine, face features deviations of specific ...
38 views

### Regression model comparison/validation in an independent cohort: other method than c-index

What I have: I developed a logistic regression model (M1) to predict lymph node metastasis in cancer patients using the variables A + B + C + D + E and a training/validation data set D1, E is the new ...
38 views

### How to deal with missing values of explanatory variables when comparing models

I want to compare several logistic regression models. The different models are built using the same initial dataset. The models differ with respect to the explanatory variables included. Many of the ...
6 views

### Model selection for panel data

I was wondering about methods that could be used to compare different model specifications when dealing with panel data. In the case of cross-sectional data, a few applicable methods would be (in my ...
49 views

### To compare two KL-divergence scores, does the prior model have to be the same for both?

The KL-divergence compares a theoretical model $p$'s distribution with the empirical model $q$'s distribution, giving a score of $0$ if they, or their information contents, are identical. Say we have ...
27 views

### How to compare multiple forecasted time series data derived from an API against actual data from an official weather station?

Pardon, I am a novice of time series analysis. I have merged together meteorological data from two different sources, one derived from the Brazilian National Institute of Meteorology (INMET) and the ...
29 views

### Significance Test for Comparing RMSE values

I want to use the RMSE measurement to compare 3 statistical regression models against a baseline model. I want to do this based on a null hypothesis testing framework where I train the models using n-...
53 views

### Can I compare lmer models with different fixed effects using anova

I know that this question sounds familiar to some other, but I believe the responses were not clear in those and were focused on REML models. I would like to know if it is sensible to compare 2 or ...
6 views

### comparing calibration across two models

I have two models that predict a binary outcome. The range of model A is [0, 0.2] while the range of model B is [0, 1] with very sparse high probability predictions. Using the typical decile binning ...
19 views

### AIC model-comparison from one level to the next

Validated community, I have an fMRI dataset that consists of 35 subjects in which I measure a signal at 450 time points at around 3000 locations (voxels) in two hemispheres. I have two competing ...
11 views

### Using forecast errors to cross-sectionally compare quality of two different data sources (using sLDA)

At a high level conceptually, I have two different data sources for panel data and I am trying to compare the quality of each data source by comparing forecast errors from models predicting an outcome ...
21 views

### Comparing linear models, when one data set is a subset of the other

There are some questions about this site on the related question of how to compare models when the datasets are completely different. For example this question, or this other question, for which only ...
61 views

### ANOVA model comparison in R: AIC, BIC, LogLik vs RSS

When comparing 2 linear models in R with anova(mod1, mod2), I used to get a nice output showing AIC, BIC, LogLik etc: ...
10 views

### getting same AIC (or any other comparison criterion values) even after using different var-cov structures when comparing GLMM models

We are comparing models that are GLMM , in which for each one of them the fixed effects are exactly the same, but in the random effects portion, we used different variance-covariance structures (i.e ...
50 views

### R lmer model: add factors or reduce factors

In mixed effect models, do you add factors one by one? Or do you reduce the factors one by one? What I am doing is as follows. Are there any problems with the steps? Build a full model: ...
75 views

### Differences between formulas for AIC and BIC

I have a question regarding the information criteria AIC and BIC: I found different formulas for the AIC/BIC, the common ones including the likelihood $\mathcal{L}$ are AIC = 2K - 2 ln(\mathcal{L})\...
104 views

### R lmer model: degree of freedom and chi square values are zero

I have built the following models: ...
42 views

### Paired t-test (or something else) to compare model performance, using a repeated train/test split?

I am looking for the correct statistical test to compare the test ROC AUC of two models. I have done the following: Randomly train/...
29 views

### Cross comparison analysis of simulation models

For my thesis, I created a simulation model which simulates the barge shipping process (basically travelling to the port and do a route where terminals are visited so that barges will be handled). I ...
17 views

### Can McNemar's Test say that Precision of one classification algorithm is better than the other?

Imagine this scenario: While presenting a new classification algo/model to my client, I asked him to run his existing algorithm (Model_Old) on 1000 objects and give me the Precision metrics, that is ...
62 views

### How do I compare cv.glmnet models with AIC?

I am using the glmnet package in R, and not(!) the caret package for my binary ElasticNet regression. I have come to the point ...
17 views

### Adjusting the number of parameters for AIC / BIC calculation in case of correlated predictors

My current understanding: Both AIC and BIC take the number of parameters as input when comparing nested models with a different number of parameters / predictors. My question: Is it necessary / a good ...
21 views

### “Best” ARMIA model changes with introduction of ARCH/GARCH errors?

When one introduces ARCH or GARCH errors into an ARIMA models, sometimes the "best" (lowest IC) will change using automated software (e.g. "auto.arima"). In a theoretical sense I ...
32 views

### Should I run a machine learning model many times?

I am comparing different classification models and for a given model (let's say logistic regression), everytime I run it, it obviously produces a slightly different value for the Accuracy and for the ...
19 views

### If two models have similar predictive power, why should we prefer the one with fewer parameters?

Was thinking a bit about model selection earlier, and I ended up getting hung up on the question: “If two models have similar predictive power, which model should I select?” For example, we often ...
7 views

### Comparing functional hypotheses accounting for uncertain interpretation of their predictions

I am interested in using an information-theoretic approach (likely AIC) to compare the explanatory power of several functional hypotheses. As an example, hypothesis H1 predicts significant association ...
19 views

### Comparing models when their performance may depend on a continuous variable

I am interested in using an information-theoretic approach (likely AIC) to compare the fit of several models to a dependent variable X. M1 may take the form ...
30 views

### Likelihood ratio not significant but bootsrapping is (and is not)

I modelled eight lmer models via the lme4 package. First, I compared the models via a likelihood ratio test yielding this: ...
11 views

### Bayesian evaluation if partitioning is justified for a dataset

I'd like to comparare whether partitioning of a dataset is justified. The data is categorical with two levels and the fitted parameter is the prevalence of positives for a certain condition in each ...
25 views

### Comparing different specifications of GARCH models with different distributional assumptions [closed]

For purely educational reasons I'm currently trying to fit different types of GARCH models, varying on the order parameters as well as flavor (standard, eGARCH, iGARCH, GJR-GARCH) and different ...
26 views

### How to compare GARCH model outcomes from two equal time series

I'm writing my thesis and will sketch the scenario I try to research: I have data for my GARCH model from two periods. The input is the same, as is the length (1y). I want to compare both the outcome ...
21 views

### Is there a way to analyse main effects on quantile mixed-effect regression in the spirit of the “ANOVA” procedure (now for medians)?

I would like to analyse my data using quantile regression with random effect. The problem is I have also categorical covariates, which will "split" the output into corresponding levels. I ...
87 views

### How to compare two LASSO models - is there an equivalent to AIC/BIC?

It is often stated online that competing OLS models explaining a common dependent variable y can be compared by calculating an AIC or BIC for each fit, and that the model with the lowest value should ...
11 views

### Adjusting p values when doing several comparisons between nested linear models

I have a linear model (in R) like this one (all variables are continous): mod0 <- lm_robust(DV ~ IV1 + IV2 + IV3 + IV4, data = df) (DV is stage of acquisition ...
21 views

### If there are several fixed effects of interest, and I want to test their inclusion via LRTs, in what order should I test them?

I have various potential fixed effect predictors for a linear mixed effect model. Some are control variables and some are predictors of interest (on their own and interactions). I am interested in ...
14 views

### What to do when a fixed effect isn't significant using LRT, but is using Satterthwaite's degrees of freedom method?

I have a linear mixed-effects model with a few control variables. There is one predictor of interest. When I compare a model with and without the effect of interest using a LRT the difference isn't ...
14 views

### Should test MSE be bootstrapped to compare fits?

Suppose you have a training and testing set. You fit two models, A and B, to the training set. They you predict on the testing set. You find (in this contrived example): Test MSE model A: 3 Test MSE ...
15 views

### what metrics can i use to compare my two classifiers?

all, i have two classifiers (xgboost and light gradient boosting) to predict if yes cancer or not. when i use roc_auc as my scoring method i get xgboost as 0.75 and light gradient boosting as 0.76. ...
70 views

### p value for difference in model outcomes

I've run two different linear mixed effects models on the same data and got two different estimates for the gradient of the longitudinal variable. e.g. model 1 has estimate 30 with standard error 5. ...
29 views

### How to assess the quality of a linear model with zero free parameters?

I have a simple linear model that predicts an outcome based on a few input variables (e.g. y = a*x + b), which are based on theory (psychology). None of the variables are free parameters, meaning ...
10 views

### Modeling slope effects to measure individual consistency

I am trying to fit a model with some pretty sparse data; I cannot collect more data so please refrain from that suggestion. I do not have a large sample size, so I have issues with singularities for ...
Most of times RMSE is used to compare models where: $RMSE=\sqrt{(\sum_i (y_{i,pred}-y_{i,obs})^2)/n}$ or namely $RMSE=(\sum_i (y_{i,pred}-y_{i,obs})^2/n)^{(1/2)}$ For some compared models, the ...