# Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

456 questions
Filter by
Sorted by
Tagged with
18 views

### Choice of Time Series Models for Forecasting Data

I have a question that to most readers here probably think is fairly simple to answer, but never the less, out of curiosity I'd like to ask: Given a time series, yt, which is stationary (say the log ...
27 views

### AIC calculation with very low negative log likelihood

I am using AIC formula (AIC=2k−2lnL) to compare different exponential models. I know that this formula is used to penalize complexed models (with high number of parameters). The problem I have is that ...
8 views

### Post-hoc analysis of neural network predictions

I have trained a model in PyTorch. My model predicts results of football games. I hypothesize that certain games in my test-set will have higher accuracy. One of the variables would be the start of ...
22 views

### AIC, pseudo-R2, or log likelihood to compare models?

I am comparing the effect of climate, across three different time brackets, on a variable. I am interested in choosing the model that best predicts the variable to answer across which timescale the ...
77 views

### How to calculate percent partial deviance explained by each predictor variable in a GAM model?

I am trying to find a sensible way to calculate the deviance explained by each predictor variable in a GAM model and need some input on my calculations. Following Simon Wood's example on the thread ...
6 views

### Does DIC favor more complicated model?

I'm doing a model selection/comparison based on two criterions - WAIC and DIC. When I consider the WAICs, my model has the smallest WAIC. However, DIC of my model is slightly bigger than DICs of few ...
21 views

I have estimated an MA(3) model and an ADL model on the differenced US unemployment rate. However, I'm in doubt as how to compare the two models? Does it make sense to compare them using information ...
13 views

### Metric to compare machine learning models with varying training data and feature set sizes

I have four machine learning models that were trained using different training sets and features. They were individually validated using the same test data and predictions were obtained. \begin{array} ...
12 views

### Compare fit of mixed model for different outcomes?

In the current literature, people are predicting X. I would like to show that the current models are more suited to predict Y than X. I have two equivalent models for the two different criteria. They ...
29 views

### The ratio of two Bayes factors for two opposite one-tailed hypotheses

I am trying to understand how Bayesian inference works, so this might be a very simple question. I have an experiment where I test two hypotheses predicting opposite results. Let’s say, hypothesis 1 (...
25 views

### Model Comparison with Regression

For a project I have to test a mediation model. I will do this with regression in R. The steps I take are the regular steps: Test if the IV has a relation with the DV (Regression) Test if the IV has ...
34 views

### random effect with one observation per group improves AIC drastically — explain

Now that I've posted an example (for a different purpose) at How will random effects with only 1 observation affect a generalized linear mixed model? (resulting in creating an account here, and being ...
22 views

### Checking for consistency and selecting the best among multiple measurement techniques

I am tasked with comparing $N$ statistical analysis methods (where $N$ is relatively small, say $N=5$), possibly not independent, for estimating some quantities $X,Y,Z$ based on some given data. Each ...
11 views

### Model Selestion - Compare AIC of data and its subset

I am trying to use AIC to assess the difference between replicates. In my experiment, each treatment has three replicates. The data from each replicates was fitted exponential model. And then combine ...
13 views

### Ranking of models on dev set does not match ranking on test

Using my train/dev/test split, I optimized four different model's hyper parameters over the development set. Suppose the ranking based on F1 score on the development set was: model 1 model 2 model 3 ...
19 views

### Can I use a (paired) t-test to compare the accuracies of classification models on a hold out test set?

When comparing the Brier scores or logloss/cross-entropy for multiple models on a hold-out test set, how can I be sure the 'best' model doesn't just have the lowest loss due to chance? When can I be ...
18 views

### Using model comparison to deal with collinearity in linear mixed models

I am working on a set given to me, which involves fitting logistic models with a couple of predictors, some of which are nearly perfectly correlated (.9), imagine, face features deviations of specific ...
40 views

### Regression model comparison/validation in an independent cohort: other method than c-index

What I have: I developed a logistic regression model (M1) to predict lymph node metastasis in cancer patients using the variables A + B + C + D + E and a training/validation data set D1, E is the new ...
39 views

### How to deal with missing values of explanatory variables when comparing models

I want to compare several logistic regression models. The different models are built using the same initial dataset. The models differ with respect to the explanatory variables included. Many of the ...
7 views

### Model selection for panel data

I was wondering about methods that could be used to compare different model specifications when dealing with panel data. In the case of cross-sectional data, a few applicable methods would be (in my ...
51 views

### To compare two KL-divergence scores, does the prior model have to be the same for both?

The KL-divergence compares a theoretical model $p$'s distribution with the empirical model $q$'s distribution, giving a score of $0$ if they, or their information contents, are identical. Say we have ...
29 views

### How to compare multiple forecasted time series data derived from an API against actual data from an official weather station?

Pardon, I am a novice of time series analysis. I have merged together meteorological data from two different sources, one derived from the Brazilian National Institute of Meteorology (INMET) and the ...
36 views

### Significance Test for Comparing RMSE values

I want to use the RMSE measurement to compare 3 statistical regression models against a baseline model. I want to do this based on a null hypothesis testing framework where I train the models using n-...
54 views

### Can I compare lmer models with different fixed effects using anova

I know that this question sounds familiar to some other, but I believe the responses were not clear in those and were focused on REML models. I would like to know if it is sensible to compare 2 or ...
6 views

### comparing calibration across two models

I have two models that predict a binary outcome. The range of model A is [0, 0.2] while the range of model B is [0, 1] with very sparse high probability predictions. Using the typical decile binning ...
20 views

### AIC model-comparison from one level to the next

Validated community, I have an fMRI dataset that consists of 35 subjects in which I measure a signal at 450 time points at around 3000 locations (voxels) in two hemispheres. I have two competing ...
11 views

### Using forecast errors to cross-sectionally compare quality of two different data sources (using sLDA)

At a high level conceptually, I have two different data sources for panel data and I am trying to compare the quality of each data source by comparing forecast errors from models predicting an outcome ...
29 views

### Comparing linear models, when one data set is a subset of the other

There are some questions about this site on the related question of how to compare models when the datasets are completely different. For example this question, or this other question, for which only ...
107 views

### ANOVA model comparison in R: AIC, BIC, LogLik vs RSS

When comparing 2 linear models in R with anova(mod1, mod2), I used to get a nice output showing AIC, BIC, LogLik etc: ...
10 views

### getting same AIC (or any other comparison criterion values) even after using different var-cov structures when comparing GLMM models

We are comparing models that are GLMM , in which for each one of them the fixed effects are exactly the same, but in the random effects portion, we used different variance-covariance structures (i.e ...
50 views

### R lmer model: add factors or reduce factors

In mixed effect models, do you add factors one by one? Or do you reduce the factors one by one? What I am doing is as follows. Are there any problems with the steps? Build a full model: ...
76 views

### Differences between formulas for AIC and BIC

I have a question regarding the information criteria AIC and BIC: I found different formulas for the AIC/BIC, the common ones including the likelihood $\mathcal{L}$ are AIC = 2K - 2 ln(\mathcal{L})\...
121 views

### R lmer model: degree of freedom and chi square values are zero

I have built the following models: ...
50 views

### Paired t-test (or something else) to compare model performance, using a repeated train/test split?

I am looking for the correct statistical test to compare the test ROC AUC of two models. I have done the following: Randomly train/...
29 views

### Cross comparison analysis of simulation models

For my thesis, I created a simulation model which simulates the barge shipping process (basically travelling to the port and do a route where terminals are visited so that barges will be handled). I ...
17 views

### Can McNemar's Test say that Precision of one classification algorithm is better than the other?

Imagine this scenario: While presenting a new classification algo/model to my client, I asked him to run his existing algorithm (Model_Old) on 1000 objects and give me the Precision metrics, that is ...
92 views

### How do I compare cv.glmnet models with AIC?

I am using the glmnet package in R, and not(!) the caret package for my binary ElasticNet regression. I have come to the point ...
18 views

### Adjusting the number of parameters for AIC / BIC calculation in case of correlated predictors

My current understanding: Both AIC and BIC take the number of parameters as input when comparing nested models with a different number of parameters / predictors. My question: Is it necessary / a good ...
21 views

### “Best” ARMIA model changes with introduction of ARCH/GARCH errors?

When one introduces ARCH or GARCH errors into an ARIMA models, sometimes the "best" (lowest IC) will change using automated software (e.g. "auto.arima"). In a theoretical sense I ...
32 views

### Should I run a machine learning model many times?

I am comparing different classification models and for a given model (let's say logistic regression), everytime I run it, it obviously produces a slightly different value for the Accuracy and for the ...
19 views

### If two models have similar predictive power, why should we prefer the one with fewer parameters?

Was thinking a bit about model selection earlier, and I ended up getting hung up on the question: “If two models have similar predictive power, which model should I select?” For example, we often ...
7 views

### Comparing functional hypotheses accounting for uncertain interpretation of their predictions

I am interested in using an information-theoretic approach (likely AIC) to compare the explanatory power of several functional hypotheses. As an example, hypothesis H1 predicts significant association ...
19 views

### Comparing models when their performance may depend on a continuous variable

I am interested in using an information-theoretic approach (likely AIC) to compare the fit of several models to a dependent variable X. M1 may take the form ...
30 views

### Likelihood ratio not significant but bootsrapping is (and is not)

I modelled eight lmer models via the lme4 package. First, I compared the models via a likelihood ratio test yielding this: ...
11 views

### Bayesian evaluation if partitioning is justified for a dataset

I'd like to comparare whether partitioning of a dataset is justified. The data is categorical with two levels and the fitted parameter is the prevalence of positives for a certain condition in each ...
27 views

### Comparing different specifications of GARCH models with different distributional assumptions [closed]

For purely educational reasons I'm currently trying to fit different types of GARCH models, varying on the order parameters as well as flavor (standard, eGARCH, iGARCH, GJR-GARCH) and different ...
26 views

### How to compare GARCH model outcomes from two equal time series

I'm writing my thesis and will sketch the scenario I try to research: I have data for my GARCH model from two periods. The input is the same, as is the length (1y). I want to compare both the outcome ...
21 views

### Is there a way to analyse main effects on quantile mixed-effect regression in the spirit of the “ANOVA” procedure (now for medians)?

I would like to analyse my data using quantile regression with random effect. The problem is I have also categorical covariates, which will "split" the output into corresponding levels. I ...