Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

Filter by
Sorted by
Tagged with
0
votes
0answers
18 views

Choice of Time Series Models for Forecasting Data

I have a question that to most readers here probably think is fairly simple to answer, but never the less, out of curiosity I'd like to ask: Given a time series, yt, which is stationary (say the log ...
1
vote
2answers
27 views

AIC calculation with very low negative log likelihood

I am using AIC formula (AIC=2k−2lnL) to compare different exponential models. I know that this formula is used to penalize complexed models (with high number of parameters). The problem I have is that ...
0
votes
0answers
8 views

Post-hoc analysis of neural network predictions

I have trained a model in PyTorch. My model predicts results of football games. I hypothesize that certain games in my test-set will have higher accuracy. One of the variables would be the start of ...
0
votes
1answer
22 views

AIC, pseudo-R2, or log likelihood to compare models?

I am comparing the effect of climate, across three different time brackets, on a variable. I am interested in choosing the model that best predicts the variable to answer across which timescale the ...
4
votes
0answers
77 views

How to calculate percent partial deviance explained by each predictor variable in a GAM model?

I am trying to find a sensible way to calculate the deviance explained by each predictor variable in a GAM model and need some input on my calculations. Following Simon Wood's example on the thread ...
0
votes
0answers
6 views

Does DIC favor more complicated model?

I'm doing a model selection/comparison based on two criterions - WAIC and DIC. When I consider the WAICs, my model has the smallest WAIC. However, DIC of my model is slightly bigger than DICs of few ...
1
vote
0answers
21 views

ARIMA vs. ADL model

I have estimated an MA(3) model and an ADL model on the differenced US unemployment rate. However, I'm in doubt as how to compare the two models? Does it make sense to compare them using information ...
0
votes
0answers
13 views

Metric to compare machine learning models with varying training data and feature set sizes

I have four machine learning models that were trained using different training sets and features. They were individually validated using the same test data and predictions were obtained. \begin{array} ...
1
vote
1answer
12 views

Compare fit of mixed model for different outcomes?

In the current literature, people are predicting X. I would like to show that the current models are more suited to predict Y than X. I have two equivalent models for the two different criteria. They ...
2
votes
1answer
29 views

The ratio of two Bayes factors for two opposite one-tailed hypotheses

I am trying to understand how Bayesian inference works, so this might be a very simple question. I have an experiment where I test two hypotheses predicting opposite results. Let’s say, hypothesis 1 (...
2
votes
1answer
25 views

Model Comparison with Regression

For a project I have to test a mediation model. I will do this with regression in R. The steps I take are the regular steps: Test if the IV has a relation with the DV (Regression) Test if the IV has ...
0
votes
0answers
34 views

random effect with one observation per group improves AIC drastically — explain

Now that I've posted an example (for a different purpose) at How will random effects with only 1 observation affect a generalized linear mixed model? (resulting in creating an account here, and being ...
0
votes
0answers
22 views

Checking for consistency and selecting the best among multiple measurement techniques

I am tasked with comparing $N$ statistical analysis methods (where $N$ is relatively small, say $N=5$), possibly not independent, for estimating some quantities $X,Y,Z$ based on some given data. Each ...
0
votes
0answers
11 views

Model Selestion - Compare AIC of data and its subset

I am trying to use AIC to assess the difference between replicates. In my experiment, each treatment has three replicates. The data from each replicates was fitted exponential model. And then combine ...
0
votes
0answers
13 views

Ranking of models on dev set does not match ranking on test

Using my train/dev/test split, I optimized four different model's hyper parameters over the development set. Suppose the ranking based on F1 score on the development set was: model 1 model 2 model 3 ...
1
vote
0answers
19 views

Can I use a (paired) t-test to compare the accuracies of classification models on a hold out test set?

When comparing the Brier scores or logloss/cross-entropy for multiple models on a hold-out test set, how can I be sure the 'best' model doesn't just have the lowest loss due to chance? When can I be ...
0
votes
0answers
18 views

Using model comparison to deal with collinearity in linear mixed models

I am working on a set given to me, which involves fitting logistic models with a couple of predictors, some of which are nearly perfectly correlated (.9), imagine, face features deviations of specific ...
1
vote
0answers
40 views

Regression model comparison/validation in an independent cohort: other method than c-index

What I have: I developed a logistic regression model (M1) to predict lymph node metastasis in cancer patients using the variables A + B + C + D + E and a training/validation data set D1, E is the new ...
3
votes
1answer
39 views

How to deal with missing values of explanatory variables when comparing models

I want to compare several logistic regression models. The different models are built using the same initial dataset. The models differ with respect to the explanatory variables included. Many of the ...
0
votes
0answers
7 views

Model selection for panel data

I was wondering about methods that could be used to compare different model specifications when dealing with panel data. In the case of cross-sectional data, a few applicable methods would be (in my ...
0
votes
0answers
51 views

To compare two KL-divergence scores, does the prior model have to be the same for both?

The KL-divergence compares a theoretical model $p$'s distribution with the empirical model $q$'s distribution, giving a score of $0$ if they, or their information contents, are identical. Say we have ...
0
votes
0answers
29 views

How to compare multiple forecasted time series data derived from an API against actual data from an official weather station?

Pardon, I am a novice of time series analysis. I have merged together meteorological data from two different sources, one derived from the Brazilian National Institute of Meteorology (INMET) and the ...
0
votes
0answers
36 views

Significance Test for Comparing RMSE values

I want to use the RMSE measurement to compare 3 statistical regression models against a baseline model. I want to do this based on a null hypothesis testing framework where I train the models using n-...
3
votes
1answer
54 views

Can I compare lmer models with different fixed effects using anova

I know that this question sounds familiar to some other, but I believe the responses were not clear in those and were focused on REML models. I would like to know if it is sensible to compare 2 or ...
0
votes
0answers
6 views

comparing calibration across two models

I have two models that predict a binary outcome. The range of model A is [0, 0.2] while the range of model B is [0, 1] with very sparse high probability predictions. Using the typical decile binning ...
0
votes
0answers
20 views

AIC model-comparison from one level to the next

Validated community, I have an fMRI dataset that consists of 35 subjects in which I measure a signal at 450 time points at around 3000 locations (voxels) in two hemispheres. I have two competing ...
0
votes
0answers
11 views

Using forecast errors to cross-sectionally compare quality of two different data sources (using sLDA)

At a high level conceptually, I have two different data sources for panel data and I am trying to compare the quality of each data source by comparing forecast errors from models predicting an outcome ...
0
votes
0answers
29 views

Comparing linear models, when one data set is a subset of the other

There are some questions about this site on the related question of how to compare models when the datasets are completely different. For example this question, or this other question, for which only ...
0
votes
2answers
107 views

ANOVA model comparison in R: AIC, BIC, LogLik vs RSS

When comparing 2 linear models in R with anova(mod1, mod2), I used to get a nice output showing AIC, BIC, LogLik etc: ...
0
votes
0answers
10 views

getting same AIC (or any other comparison criterion values) even after using different var-cov structures when comparing GLMM models

We are comparing models that are GLMM , in which for each one of them the fixed effects are exactly the same, but in the random effects portion, we used different variance-covariance structures (i.e ...
3
votes
1answer
50 views

R lmer model: add factors or reduce factors

In mixed effect models, do you add factors one by one? Or do you reduce the factors one by one? What I am doing is as follows. Are there any problems with the steps? Build a full model: ...
1
vote
1answer
76 views

Differences between formulas for AIC and BIC

I have a question regarding the information criteria AIC and BIC: I found different formulas for the AIC/BIC, the common ones including the likelihood $\mathcal{L}$ are $$AIC = 2K - 2 ln(\mathcal{L})\...
3
votes
1answer
121 views

R lmer model: degree of freedom and chi square values are zero

I have built the following models: ...
0
votes
1answer
50 views

Paired t-test (or something else) to compare model performance, using a repeated train/test split?

I am looking for the correct statistical test to compare the test ROC AUC of two models. I have done the following: Randomly train/...
0
votes
0answers
29 views

Cross comparison analysis of simulation models

For my thesis, I created a simulation model which simulates the barge shipping process (basically travelling to the port and do a route where terminals are visited so that barges will be handled). I ...
0
votes
0answers
17 views

Can McNemar's Test say that Precision of one classification algorithm is better than the other?

Imagine this scenario: While presenting a new classification algo/model to my client, I asked him to run his existing algorithm (Model_Old) on 1000 objects and give me the Precision metrics, that is ...
0
votes
1answer
92 views

How do I compare cv.glmnet models with AIC?

I am using the glmnet package in R, and not(!) the caret package for my binary ElasticNet regression. I have come to the point ...
1
vote
1answer
18 views

Adjusting the number of parameters for AIC / BIC calculation in case of correlated predictors

My current understanding: Both AIC and BIC take the number of parameters as input when comparing nested models with a different number of parameters / predictors. My question: Is it necessary / a good ...
1
vote
0answers
21 views

“Best” ARMIA model changes with introduction of ARCH/GARCH errors?

When one introduces ARCH or GARCH errors into an ARIMA models, sometimes the "best" (lowest IC) will change using automated software (e.g. "auto.arima"). In a theoretical sense I ...
1
vote
2answers
32 views

Should I run a machine learning model many times?

I am comparing different classification models and for a given model (let's say logistic regression), everytime I run it, it obviously produces a slightly different value for the Accuracy and for the ...
1
vote
0answers
19 views

If two models have similar predictive power, why should we prefer the one with fewer parameters?

Was thinking a bit about model selection earlier, and I ended up getting hung up on the question: “If two models have similar predictive power, which model should I select?” For example, we often ...
0
votes
0answers
7 views

Comparing functional hypotheses accounting for uncertain interpretation of their predictions

I am interested in using an information-theoretic approach (likely AIC) to compare the explanatory power of several functional hypotheses. As an example, hypothesis H1 predicts significant association ...
1
vote
0answers
19 views

Comparing models when their performance may depend on a continuous variable

I am interested in using an information-theoretic approach (likely AIC) to compare the fit of several models to a dependent variable X. M1 may take the form ...
0
votes
1answer
30 views

Likelihood ratio not significant but bootsrapping is (and is not)

I modelled eight lmer models via the lme4 package. First, I compared the models via a likelihood ratio test yielding this: ...
1
vote
1answer
11 views

Bayesian evaluation if partitioning is justified for a dataset

I'd like to comparare whether partitioning of a dataset is justified. The data is categorical with two levels and the fitted parameter is the prevalence of positives for a certain condition in each ...
2
votes
1answer
27 views

Comparing different specifications of GARCH models with different distributional assumptions [closed]

For purely educational reasons I'm currently trying to fit different types of GARCH models, varying on the order parameters as well as flavor (standard, eGARCH, iGARCH, GJR-GARCH) and different ...
1
vote
1answer
26 views

How to compare GARCH model outcomes from two equal time series

I'm writing my thesis and will sketch the scenario I try to research: I have data for my GARCH model from two periods. The input is the same, as is the length (1y). I want to compare both the outcome ...
2
votes
0answers
21 views

Is there a way to analyse main effects on quantile mixed-effect regression in the spirit of the “ANOVA” procedure (now for medians)?

I would like to analyse my data using quantile regression with random effect. The problem is I have also categorical covariates, which will "split" the output into corresponding levels. I ...
1
vote
0answers
111 views

How to compare two LASSO models - is there an equivalent to AIC/BIC?

It is often stated online that competing OLS models explaining a common dependent variable y can be compared by calculating an AIC or BIC for each fit, and that the model with the lowest value should ...
0
votes
0answers
11 views

Adjusting p values when doing several comparisons between nested linear models

I have a linear model (in R) like this one (all variables are continous): mod0 <- lm_robust(DV ~ IV1 + IV2 + IV3 + IV4, data = df) (DV is stage of acquisition ...

1
2 3 4 5
10