Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

Filter by
Sorted by
Tagged with
1
vote
1answer
20 views

The ratio of two Bayes factors for two opposite one-tailed hypotheses

I am trying to understand how Bayesian inference works, so this might be a very simple question. I have an experiment where I test two hypotheses predicting opposite results. Let’s say, hypothesis 1 (...
2
votes
1answer
21 views

Model Comparison with Regression

For a project I have to test a mediation model. I will do this with regression in R. The steps I take are the regular steps: Test if the IV has a relation with the DV (Regression) Test if the IV has ...
0
votes
0answers
27 views

random effect with one observation per group improves AIC drastically — explain

Now that I've posted an example (for a different purpose) at How will random effects with only 1 observation affect a generalized linear mixed model? (resulting in creating an account here, and being ...
0
votes
0answers
20 views

Checking for consistency and selecting the best among multiple measurement techniques

I am tasked with comparing $N$ statistical analysis methods (where $N$ is relatively small, say $N=5$), possibly not independent, for estimating some quantities $X,Y,Z$ based on some given data. Each ...
0
votes
0answers
11 views

Model Selestion - Compare AIC of data and its subset

I am trying to use AIC to assess the difference between replicates. In my experiment, each treatment has three replicates. The data from each replicates was fitted exponential model. And then combine ...
0
votes
0answers
13 views

Ranking of models on dev set does not match ranking on test

Using my train/dev/test split, I optimized four different model's hyper parameters over the development set. Suppose the ranking based on F1 score on the development set was: model 1 model 2 model 3 ...
1
vote
0answers
18 views

Can I use a (paired) t-test to compare the accuracies of classification models on a hold out test set?

When comparing the Brier scores or logloss/cross-entropy for multiple models on a hold-out test set, how can I be sure the 'best' model doesn't just have the lowest loss due to chance? When can I be ...
0
votes
0answers
18 views

Using model comparison to deal with collinearity in linear mixed models

I am working on a set given to me, which involves fitting logistic models with a couple of predictors, some of which are nearly perfectly correlated (.9), imagine, face features deviations of specific ...
1
vote
0answers
38 views

Regression model comparison/validation in an independent cohort: other method than c-index

What I have: I developed a logistic regression model (M1) to predict lymph node metastasis in cancer patients using the variables A + B + C + D + E and a training/validation data set D1, E is the new ...
3
votes
1answer
38 views

How to deal with missing values of explanatory variables when comparing models

I want to compare several logistic regression models. The different models are built using the same initial dataset. The models differ with respect to the explanatory variables included. Many of the ...
0
votes
0answers
6 views

Model selection for panel data

I was wondering about methods that could be used to compare different model specifications when dealing with panel data. In the case of cross-sectional data, a few applicable methods would be (in my ...
0
votes
0answers
49 views

To compare two KL-divergence scores, does the prior model have to be the same for both?

The KL-divergence compares a theoretical model $p$'s distribution with the empirical model $q$'s distribution, giving a score of $0$ if they, or their information contents, are identical. Say we have ...
0
votes
0answers
27 views

How to compare multiple forecasted time series data derived from an API against actual data from an official weather station?

Pardon, I am a novice of time series analysis. I have merged together meteorological data from two different sources, one derived from the Brazilian National Institute of Meteorology (INMET) and the ...
0
votes
0answers
29 views

Significance Test for Comparing RMSE values

I want to use the RMSE measurement to compare 3 statistical regression models against a baseline model. I want to do this based on a null hypothesis testing framework where I train the models using n-...
3
votes
1answer
53 views

Can I compare lmer models with different fixed effects using anova

I know that this question sounds familiar to some other, but I believe the responses were not clear in those and were focused on REML models. I would like to know if it is sensible to compare 2 or ...
0
votes
0answers
6 views

comparing calibration across two models

I have two models that predict a binary outcome. The range of model A is [0, 0.2] while the range of model B is [0, 1] with very sparse high probability predictions. Using the typical decile binning ...
0
votes
0answers
19 views

AIC model-comparison from one level to the next

Validated community, I have an fMRI dataset that consists of 35 subjects in which I measure a signal at 450 time points at around 3000 locations (voxels) in two hemispheres. I have two competing ...
0
votes
0answers
11 views

Using forecast errors to cross-sectionally compare quality of two different data sources (using sLDA)

At a high level conceptually, I have two different data sources for panel data and I am trying to compare the quality of each data source by comparing forecast errors from models predicting an outcome ...
0
votes
0answers
21 views

Comparing linear models, when one data set is a subset of the other

There are some questions about this site on the related question of how to compare models when the datasets are completely different. For example this question, or this other question, for which only ...
0
votes
2answers
61 views

ANOVA model comparison in R: AIC, BIC, LogLik vs RSS

When comparing 2 linear models in R with anova(mod1, mod2), I used to get a nice output showing AIC, BIC, LogLik etc: ...
0
votes
0answers
10 views

getting same AIC (or any other comparison criterion values) even after using different var-cov structures when comparing GLMM models

We are comparing models that are GLMM , in which for each one of them the fixed effects are exactly the same, but in the random effects portion, we used different variance-covariance structures (i.e ...
3
votes
1answer
50 views

R lmer model: add factors or reduce factors

In mixed effect models, do you add factors one by one? Or do you reduce the factors one by one? What I am doing is as follows. Are there any problems with the steps? Build a full model: ...
1
vote
1answer
75 views

Differences between formulas for AIC and BIC

I have a question regarding the information criteria AIC and BIC: I found different formulas for the AIC/BIC, the common ones including the likelihood $\mathcal{L}$ are $$AIC = 2K - 2 ln(\mathcal{L})\...
3
votes
1answer
104 views

R lmer model: degree of freedom and chi square values are zero

I have built the following models: ...
0
votes
1answer
42 views

Paired t-test (or something else) to compare model performance, using a repeated train/test split?

I am looking for the correct statistical test to compare the test ROC AUC of two models. I have done the following: Randomly train/...
0
votes
0answers
29 views

Cross comparison analysis of simulation models

For my thesis, I created a simulation model which simulates the barge shipping process (basically travelling to the port and do a route where terminals are visited so that barges will be handled). I ...
0
votes
0answers
17 views

Can McNemar's Test say that Precision of one classification algorithm is better than the other?

Imagine this scenario: While presenting a new classification algo/model to my client, I asked him to run his existing algorithm (Model_Old) on 1000 objects and give me the Precision metrics, that is ...
0
votes
1answer
62 views

How do I compare cv.glmnet models with AIC?

I am using the glmnet package in R, and not(!) the caret package for my binary ElasticNet regression. I have come to the point ...
1
vote
1answer
17 views

Adjusting the number of parameters for AIC / BIC calculation in case of correlated predictors

My current understanding: Both AIC and BIC take the number of parameters as input when comparing nested models with a different number of parameters / predictors. My question: Is it necessary / a good ...
1
vote
0answers
21 views

“Best” ARMIA model changes with introduction of ARCH/GARCH errors?

When one introduces ARCH or GARCH errors into an ARIMA models, sometimes the "best" (lowest IC) will change using automated software (e.g. "auto.arima"). In a theoretical sense I ...
1
vote
2answers
32 views

Should I run a machine learning model many times?

I am comparing different classification models and for a given model (let's say logistic regression), everytime I run it, it obviously produces a slightly different value for the Accuracy and for the ...
1
vote
0answers
19 views

If two models have similar predictive power, why should we prefer the one with fewer parameters?

Was thinking a bit about model selection earlier, and I ended up getting hung up on the question: “If two models have similar predictive power, which model should I select?” For example, we often ...
0
votes
0answers
7 views

Comparing functional hypotheses accounting for uncertain interpretation of their predictions

I am interested in using an information-theoretic approach (likely AIC) to compare the explanatory power of several functional hypotheses. As an example, hypothesis H1 predicts significant association ...
1
vote
0answers
19 views

Comparing models when their performance may depend on a continuous variable

I am interested in using an information-theoretic approach (likely AIC) to compare the fit of several models to a dependent variable X. M1 may take the form ...
0
votes
1answer
30 views

Likelihood ratio not significant but bootsrapping is (and is not)

I modelled eight lmer models via the lme4 package. First, I compared the models via a likelihood ratio test yielding this: ...
1
vote
1answer
11 views

Bayesian evaluation if partitioning is justified for a dataset

I'd like to comparare whether partitioning of a dataset is justified. The data is categorical with two levels and the fitted parameter is the prevalence of positives for a certain condition in each ...
2
votes
1answer
25 views

Comparing different specifications of GARCH models with different distributional assumptions [closed]

For purely educational reasons I'm currently trying to fit different types of GARCH models, varying on the order parameters as well as flavor (standard, eGARCH, iGARCH, GJR-GARCH) and different ...
1
vote
1answer
26 views

How to compare GARCH model outcomes from two equal time series

I'm writing my thesis and will sketch the scenario I try to research: I have data for my GARCH model from two periods. The input is the same, as is the length (1y). I want to compare both the outcome ...
2
votes
0answers
21 views

Is there a way to analyse main effects on quantile mixed-effect regression in the spirit of the “ANOVA” procedure (now for medians)?

I would like to analyse my data using quantile regression with random effect. The problem is I have also categorical covariates, which will "split" the output into corresponding levels. I ...
1
vote
0answers
87 views

How to compare two LASSO models - is there an equivalent to AIC/BIC?

It is often stated online that competing OLS models explaining a common dependent variable y can be compared by calculating an AIC or BIC for each fit, and that the model with the lowest value should ...
0
votes
0answers
11 views

Adjusting p values when doing several comparisons between nested linear models

I have a linear model (in R) like this one (all variables are continous): mod0 <- lm_robust(DV ~ IV1 + IV2 + IV3 + IV4, data = df) (DV is stage of acquisition ...
5
votes
1answer
21 views

If there are several fixed effects of interest, and I want to test their inclusion via LRTs, in what order should I test them?

I have various potential fixed effect predictors for a linear mixed effect model. Some are control variables and some are predictors of interest (on their own and interactions). I am interested in ...
0
votes
0answers
14 views

What to do when a fixed effect isn't significant using LRT, but is using Satterthwaite's degrees of freedom method?

I have a linear mixed-effects model with a few control variables. There is one predictor of interest. When I compare a model with and without the effect of interest using a LRT the difference isn't ...
1
vote
1answer
14 views

Should test MSE be bootstrapped to compare fits?

Suppose you have a training and testing set. You fit two models, A and B, to the training set. They you predict on the testing set. You find (in this contrived example): Test MSE model A: 3 Test MSE ...
0
votes
0answers
15 views

what metrics can i use to compare my two classifiers?

all, i have two classifiers (xgboost and light gradient boosting) to predict if yes cancer or not. when i use roc_auc as my scoring method i get xgboost as 0.75 and light gradient boosting as 0.76. ...
2
votes
1answer
70 views

p value for difference in model outcomes

I've run two different linear mixed effects models on the same data and got two different estimates for the gradient of the longitudinal variable. e.g. model 1 has estimate 30 with standard error 5. ...
0
votes
1answer
29 views

How to assess the quality of a linear model with zero free parameters?

I have a simple linear model that predicts an outcome based on a few input variables (e.g. y = a*x + b), which are based on theory (psychology). None of the variables are free parameters, meaning ...
0
votes
0answers
10 views

Modeling slope effects to measure individual consistency

I am trying to fit a model with some pretty sparse data; I cannot collect more data so please refrain from that suggestion. I do not have a large sample size, so I have issues with singularities for ...
1
vote
1answer
29 views

RMSE(Root mean square error) value as a model comparison method

Most of times RMSE is used to compare models where: $RMSE=\sqrt{(\sum_i (y_{i,pred}-y_{i,obs})^2)/n}$ or namely $RMSE=(\sum_i (y_{i,pred}-y_{i,obs})^2/n)^{(1/2)}$ For some compared models, the ...
1
vote
1answer
12 views

How to test two methods for their capability of quantifiy target?

I have two quantitative methods A and B, and I ran a series of measurements against a known quantity of target. The difference from the expectation was 56 and 13. To check which method is truly ...

1
2 3 4 5
9