Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

Filter by
Sorted by
Tagged with
0
votes
0answers
15 views

Passive Combination of two Classifiers at the level of Class Labels

I have three classifiers - 1. Vision based classifier trained to detect class labels such as pedestrian and cars 2. Radar based classifier to detect same class labels 3. Lidar based classifiers to ...
1
vote
1answer
40 views

Model selection based on the closest fit to another model

For each subject I have two measures, let's say $X$ and $Y$, that correspond to the same measurement but each from a different measurement device. In theory we assume that ideally these two ...
2
votes
0answers
55 views

Why is pseudo-Bayesian Model Averaging using WAIC giving counterintuitive results for my model? Issue with unconstrained data uncertainty?

I'm currently working on a model which fits a pair of curves which are parameterized by some shape and scaling parameters. I want to produce weights for two different models. In my first model, which ...
0
votes
0answers
44 views

Is there an AIC and BIC equivalent for MAP?

I would like to know if there is an AIC or BIC equivalent for maximum a posterior estimation. I'm trying to compare several different models that have been fit using MAP, but I am unsure of the best ...
3
votes
1answer
223 views

Alternatives for Diebold-Mariano test when comparing the best forecast among many against a benchmark

Suppose I encounter a new forecasting method and I wish to evaluate it against a benchmark. I can obtain forecasts from the two methods and compare them to actual realizations and thus obtain the ...
1
vote
1answer
293 views

Dietterich's 5x2cv paired t-test for regression problems

In 1998 Dietterich proposed an algorithm for comparing classification algorithms using a series of cross validated t-tests: Dietterich, Thomas G. "Approximate statistical tests for comparing ...
1
vote
0answers
129 views

K-Fold Cross Validation, the right way

I searched for this in many forums, but didn't find any answers that would answer my question (or I didn't understand correctly). So, I will post here. I have a dataset A I have a machine learning ...
0
votes
1answer
84 views

Is the AIC of an ARIMA model comparable with the AIC of a regression with ARIMA errors?

I already know that models from different "families" cannot be compared through AIC (or other information criteria such as BIC or HQIC), however, I'm not sure about the specific case of ARIMA and ...
0
votes
1answer
75 views

Nested Cross-Validation with Test Set (Model Selection)

I have seen many examples on Nested CV where someone takes the entire dataset and performs nested CV on it. My question is: for model comparison shouldn't we first split the original dataset into ...
2
votes
1answer
41 views

What are some of the most correct/accepted ways to tune and compare different models in an academic context?

Those days, I have been reviewing different academic papers which mainly compare the performance of different machine learning methods on a particular problem. And I was surprised by the variety of ...
2
votes
1answer
309 views

Is it valid to use Anova (in R) to compare alternative multinomial log-linear models?

I am familiar with the idea of comparing alternative linear regression models using anova(model1,model2), for models fitted using ...
0
votes
0answers
25 views

Choosing the best aggregation location measure

I'm trying to analyze a granular dataset that is composed as follows: ...
2
votes
0answers
148 views

WAIC for model comparisons--overly conservative?

I'm having a hard time wrapping my head around the relationship between model posterior predictions and model comparisons via WAIC. Specifically, how do I interpret findings where a model including ...
4
votes
2answers
271 views

Should I cross-validate metrics that were not optimised?

I want to compare two models. Say I have two objective functions on the same data $f(X,y,\theta)$ and $g(X,y,\theta)$ that both evaluate the models performance in ways that I am interested in ($\theta$...
1
vote
1answer
288 views

Can I use the AIC to compare ordered logistic regression models based on different datasets?

I performed ordered logistic regression using the polr() command in R. I did this with the same independent and dependent variable (the X and Y are the same for every model), but the exact data in ...
1
vote
0answers
32 views

DIC to compare models with different numbers of parameter?

I am interested in comparing hierarchical Bayesian models based on the same dataset but differing in their spatial and temporal resolution. In short, I am looking at animal population changes over ...
1
vote
1answer
56 views

Comparing two machine learning models

Let's say I'm working with two models, ResNet-50 and ResNext-50, and I want to compare the results. Due to the stochastic nature of deep learning, would it be advantageous or even encouraged to seed ...
1
vote
0answers
40 views

Loss function for one-step-ahead volatility forcasts

I'm trying to perform the MCS test using the R-package "MCS" to compare GARCH-MIDAS Models. The loss function requires as inputs a vector with some realized volatility measure ˜ σt+1 (I chose the ...
0
votes
0answers
31 views

Likelihood ratio to quantify the similarity between one sample with two other matched samples

I conducted a study with 3 conditions and N subjects. All subjects performed all conditions once. I would like to know if the first condition is similar to the second or third condition. Formally, ...
5
votes
2answers
150 views

Bayesian methods are about averaging over uncertainty rather than optimization. Explain?

I came across the statement "The key ingredient in Bayesian methods is to average over your uncertain variables and parameters, rather than to optimize". Can someone explain why this is? ...
0
votes
2answers
305 views

Bayes factors and predictive accuracy in model comparison in rstan / brms

Despite reading up on the subject, I can't wrap my head round it, so the question remains on shaky grounds, and responses along the lines of "read chapter x" are very welcome. What I'm doing is I'm ...
1
vote
0answers
566 views

NLS model comparisons by ANOVA

I am comparing NLS models from two groups, males and females. I compared the models by ANOVAs, but I am hesitating which is the correct model comparison to use: a) Subgroup Males vs Subgroup Females ...
1
vote
2answers
45 views

How to compare two classifications, visually?

I have two classifiers, and I was asked (for learning purposes) to compare their predictions, visually. The target value is a real value (sentiment) What could be a good way to compare them?
3
votes
2answers
154 views

Permutation test comparing nested non-linear models with an exchangeable dummy variable

This question is closely related to an earlier question, but I realized my case was actually a lot more specific than the way I formulated it there, in ways that I think merit a separate answer. I ...
3
votes
1answer
677 views

Interpretation of R output from Cohen's Kappa

I have the following result from carrying out Cohen's kappa in R ...
0
votes
1answer
424 views

How to interpret/compare R2 score(s)? [duplicate]

I know that an R² score of 1 is a perfect fit of the model to the truth, a 0 is an constant output regardless of the input, and that negative values are possible when the output varies, but there is ...
1
vote
0answers
63 views

AIC for latent variable models

I'm trying to use BIC/AIC for model comparison and want to know what the number of parameters is. The models I'm unsure about are linear Gaussian state space models with nonlinear observations. ...
0
votes
1answer
73 views

Can I compare AIC values for similar models fitted for two samples from the same population?

If I have two samples (n=300) from the same population and I'm fitting a GLMM (Generalised Linear Mixed-effects Model) with a similar response and explanatory variables but with a completely different ...
1
vote
0answers
20 views

Which is the more reliable method for reporting classification results in deep learning?

I have two methods to compare, the one which reports weighted F1-score with imbalanced data and, the one which reports better F1-score with balanced data. I am confused as to which method's ...
2
votes
1answer
128 views

Compare two mixed-effect models differing on the nested effect

I need to compare two models with repeated measures (I'm using using lme4 or glmmTMB for the random effects) ...
0
votes
1answer
126 views

Interpreting ANOVA f-test for model comparison

In the book ISLR, there's a lab activity which compares the nested models. ...
7
votes
2answers
1k views

Logistic regression BIC: what's the right N?

TL;DR: Which $N$ is correct for BIC in logistic regression, the aggregated binomial or Bernoulli $N$? UPDATES AT BOTTOM Suppose I have a data set to which I'd like to apply logistic regression. For ...
0
votes
0answers
630 views

BIC score for Gaussian Mixture Model

I'm not sure how to compute a BIC score for multiple classes. For exmaple , I have a supervised problem with 3 classes.I fit 3 gaussian using MLE. Then, if I want to compute BIC score: I have to ...
2
votes
0answers
130 views

How do you / can you compare Bayesian vs. frequentist regression models?

I am working on a regression model to predict a target variable in a dataset with over 100 features. Three different regression models are defined and fit in order to compare their performance using $...
8
votes
2answers
190 views

How can I compare models without fitting?

Regression and machine learning are used in the natural sciences to test hypotheses, estimate parameters, and make predictions by fitting models to data. However, when I have an a priori model, I don'...
0
votes
0answers
66 views

compare the outputs of two different ML models

Let's imagine I have two completely different multi-class ML models, let's call them ML1 and ML2. The models were trained on completely different data with different target classes. As an output, the ...
1
vote
0answers
68 views

Using unnormalized log likelihood for model comparison with different features?

Is it possible to directly use the log likelihood from fitting a model, for the purpose of model comparison? For example, if I'm using a logistic regression model, and I want to see if adding ...
2
votes
0answers
252 views

How to compare non-nested models (with different number of items) in CFA?

I investigate the factor structur of an existing scale in a data set with N = 714 (using lavaan, R). In this context I would like to compare to alternative models: Model 1: Two factors with 5 items ...
3
votes
1answer
329 views

Bootstrap Hypothesis Test: Comparing the Performance of two Models

I have two models $A$ and $B$ as well as a training set $X$. I want to test whether there is a significant difference in performance between model $A$ and $B$. I'm attempting to do this via ...
4
votes
0answers
211 views

Is the use of loglik or AIC to compare logit/probit/cloglog models valid?

I would like to know whether I can use AIC, or if the models have the same number of predictors, the log-likelihood, to compare logit vs probit vs cloglog models (fitted for instance with glmer or ...
1
vote
0answers
51 views

Comparing multiple hurdle to compensatory scoring (of tests)

I plan to do both Monte Carlo and real world research to compare the efficacy of three approaches to combining test scores: (1) fully compensatory scoring, (2) multiple hurdle, and (3) a combination ...
0
votes
1answer
41 views

Predicting proportion of individuals belonging to different classes

I am trying to compare performances of different classifiers to predict some data. The variable I wand to predict is binary (A or B), and I have a bunch of predictive variables to predict to what ...
2
votes
1answer
180 views

summary of lm to compare models instead of ANOVA

What does it mean to use anova to compare models? For example, if I have two models $$ \text{model}_1 \colon y = b_0 + b_1 x_1 + b_2 x_2 $$ and $$ \text{...
1
vote
0answers
19 views

Model comparison for predictors with differently shaped distributions

I want to use model comparison to evaluate the claim "Predictor A outperforms Predictor B in modeling Dependent X". The (potential) problem is that within my sample, Predictor A and Predictor B have ...
1
vote
1answer
26 views

Compare a real-world process with model prediction

I have a real-world time series process with an dependent variable Y responds to multiple independent variables (not necessary a linear response but can assume so if necessary). A non-statistical ...
0
votes
0answers
237 views

Comparing the fit of quasi-Poisson and negative binomial models

Is there any way to compare the fit of quasi-Poission and negative binomial models? If so, can it be done in R?
2
votes
1answer
488 views

Testing the significance of a main effect using model comparison, when an interaction is included

I am fitting a linear mixed effect model (using lmer), and want to obtain p-values via model comparison. As far as I understand it, the procedure is: (i) create a ...
1
vote
0answers
85 views

Comparing mixed effects models

I have 2 dependent variables and 1 explanatory variable of interest (there will be other control variables). I have a data structure requiring crossed random effects, so will be running a mixed ...
2
votes
1answer
57 views

Specifying multilevel model structure when random effects exhaust the population

I have been working with a dataset featuring observations at the county level for about 1300 of the ~3100 or so counties of the United States. These 1300 counties are drawn from every state in the ...
0
votes
1answer
54 views

Why compare measures of dispersion with ratios?

I'm writing a paper on a novel statistical model estimated using MCMC and am currently evaluating it using simulated data. We are comparing the performance of our model to an established model as ...

1 2
3
4 5
10