Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

0
votes
0answers
21 views

Likelihood ratio to quantify the similarity between one sample with two other matched samples

I conducted a study with 3 conditions and N subjects. All subjects performed all conditions once. I would like to know if the first condition is similar to the second or third condition. Formally, ...
5
votes
2answers
122 views

Bayesian methods are about averaging over uncertainty rather than optimization. Explain?

I came across the statement "The key ingredient in Bayesian methods is to average over your uncertain variables and parameters, rather than to optimize". Can someone explain why this is? ...
0
votes
2answers
29 views

Bayes factors and predictive accuracy in model comparison in rstan / brms

Despite reading up on the subject, I can't wrap my head round it, so the question remains on shaky grounds, and responses along the lines of "read chapter x" are very welcome. What I'm doing is I'm ...
1
vote
0answers
26 views

NLS model comparisons by ANOVA

I am comparing NLS models from two groups, males and females. I compared the models by ANOVAs, but I am hesitating which is the correct model comparison to use: a) Subgroup Males vs Subgroup Females ...
0
votes
0answers
10 views

Deterioration of the accuracy of a system over time

I have a system that compares the predicted variable with the true variable by calculating the absolute error percentages. . Where $\pi$ is the predicted variable and P is the true variable. And you ...
1
vote
2answers
24 views

How to compare two classifications, visually?

I have two classifiers, and I was asked (for learning purposes) to compare their predictions, visually. The target value is a real value (sentiment) What could be a good way to compare them?
0
votes
0answers
6 views

Marginal effect from two models with different dependent variables

This question is somehow related to this one and this one. I have two models with the same independent variables on the same dataset. Model1: Y1 = alpha0 + alpha1 X1 + alpha2 X2 + error1 Model2: ...
4
votes
2answers
100 views

Permutation test comparing nested non-linear models with an exchangeable dummy variable

This question is closely related to an earlier question, but I realized my case was actually a lot more specific than the way I formulated it there, in ways that I think merit a separate answer. I ...
0
votes
0answers
10 views

Model comparison with new IV over data subset: to rebuild base model or not

I'm modeling a few DVs with a set of IVs via glmer. I have successfully built a drop-1 nested model comparison framework to test the significance of each variable. ...
3
votes
1answer
103 views

Interpretation of R output from Cohen's Kappa

I have the following result from carrying out Cohen's kappa in R ...
1
vote
0answers
29 views

AIC for latent variable models

I'm trying to use BIC/AIC for model comparison and want to know what the number of parameters is. The models I'm unsure about are linear Gaussian state space models with nonlinear observations. ...
0
votes
1answer
44 views

Can I compare AIC values for similar models fitted for two samples from the same population?

If I have two samples (n=300) from the same population and I'm fitting a GLMM (Generalised Linear Mixed-effects Model) with a similar response and explanatory variables but with a completely different ...
1
vote
0answers
12 views

Which is the more reliable method for reporting classification results in deep learning?

I have two methods to compare, the one which reports weighted F1-score with imbalanced data and, the one which reports better F1-score with balanced data. I am confused as to which method's ...
2
votes
1answer
37 views

Compare two mixed-effect models differing on the nested effect

I need to compare two models with repeated measures (I'm using using lme4 or glmmTMB for the random effects) ...
0
votes
1answer
30 views

Interpreting ANOVA f-test for model comparison

In the book ISLR, there's a lab activity which compares the nested models. ...
7
votes
2answers
289 views

Logistic regression BIC: what's the right N?

TL;DR: Which $N$ is correct for BIC in logistic regression, the aggregated binomial or Bernoulli $N$? UPDATES AT BOTTOM Suppose I have a data set to which I'd like to apply logistic regression. For ...
0
votes
0answers
49 views

BIC score for Gaussian Mixture Model

I'm not sure how to compute a BIC score for multiple classes. For exmaple , I have a supervised problem with 3 classes.I fit 3 gaussian using MLE. Then, if I want to compute BIC score: I have to ...
0
votes
0answers
16 views

Metrics for comparing a multi-class model vs. a multi-label model?

The dataset is a multi-label dataset, where each item can have more than one labels. I first trained a multi-class classifier by randomly select the label for each item at each iteration, and ...
1
vote
0answers
57 views

How do you / can you compare Bayesian vs. frequentist regression models?

I am working on a regression model to predict a target variable in a dataset with over 100 features. Three different regression models are defined and fit in order to compare their performance using $...
1
vote
0answers
28 views

Model comparison in R

I want to stack three models (RF, SVM, ANN) using Stochastic Gradient Boosting as the meta-learner. After computing predictions from my sub-models, I used the ModelCor function in R to compute a model ...
0
votes
0answers
48 views

Kolmogorov Smirnov, CVM, AD all reject Null hypothesis - comparing two samples

I have an empirical sample (with N >10000) and I fit it to a beta-distribution. Now, as a goodness of fit, I first looked at the density function plots of the empirical and the estimated beta ...
8
votes
2answers
160 views

How can I compare models without fitting?

Regression and machine learning are used in the natural sciences to test hypotheses, estimate parameters, and make predictions by fitting models to data. However, when I have an a priori model, I don'...
0
votes
0answers
22 views

compare the outputs of two different ML models

Let's imagine I have two completely different multi-class ML models, let's call them ML1 and ML2. The models were trained on completely different data with different target classes. As an output, the ...
1
vote
0answers
31 views

Using unnormalized log likelihood for model comparison with different features?

Is it possible to directly use the log likelihood from fitting a model, for the purpose of model comparison? For example, if I'm using a logistic regression model, and I want to see if adding ...
2
votes
0answers
133 views

How to compare non-nested models (with different number of items) in CFA?

I investigate the factor structur of an existing scale in a data set with N = 714 (using lavaan, R). In this context I would like to compare to alternative models: Model 1: Two factors with 5 items ...
2
votes
1answer
72 views

Bootstrap Hypothesis Test: Comparing the Performance of two Models

I have two models $A$ and $B$ as well as a training set $X$. I want to test whether there is a significant difference in performance between model $A$ and $B$. I'm attempting to do this via ...
3
votes
0answers
93 views

Is the use of loglik or AIC to compare logit/probit/cloglog models valid?

I would like to know whether I can use AIC, or if the models have the same number of predictors, the log-likelihood, to compare logit vs probit vs cloglog models (fitted for instance with glmer or ...
0
votes
0answers
24 views

Comparing multiple hurdle to compensatory scoring (of tests)

I plan to do both Monte Carlo and real world research to compare the efficacy of three approaches to combining test scores: (1) fully compensatory scoring, (2) multiple hurdle, and (3) a combination ...
0
votes
1answer
32 views

Predicting proportion of individuals belonging to different classes

I am trying to compare performances of different classifiers to predict some data. The variable I wand to predict is binary (A or B), and I have a bunch of predictive variables to predict to what ...
0
votes
0answers
10 views

Comparing two Models gained from process discovery

I have a rather basic question on process mining that I however hardly get answered: currently, I do some research on regulated learning in educational/psychological research: I already drew up two ...
0
votes
0answers
59 views

Model comparison when using Harrell's bootstrap-based optimism-correction Internal Validation procedure

I have 2 non-nested models that I would like to compare with respect to their predictive accuracy (concordance index). I have used Harrell's (Efron-Gong) bootstrap-based optimism-correction internal ...
2
votes
1answer
113 views

summary of lm to compare models instead of ANOVA

What does it mean to use anova to compare models? For example, if I have two models $$ \text{model}_1 \colon y = b_0 + b_1 x_1 + b_2 x_2 $$ and $$ \text{...
1
vote
0answers
15 views

Model comparison for predictors with differently shaped distributions

I want to use model comparison to evaluate the claim "Predictor A outperforms Predictor B in modeling Dependent X". The (potential) problem is that within my sample, Predictor A and Predictor B have ...
0
votes
0answers
12 views

Checking equality of coefficients from two different count data regression models

Suppose, I have cluster survey data. Individuals are nested within cluster. My outcome variable of interests are two count variables Y1 and Y2. I have a set of predictor variables X1, X2, ... X5. Now ...
1
vote
1answer
21 views

Compare a real-world process with model prediction

I have a real-world time series process with an dependent variable Y responds to multiple independent variables (not necessary a linear response but can assume so if necessary). A non-statistical ...
0
votes
0answers
86 views

Compairing the fit of quasi-Poisson and negative binomial models

Is there any way to compare the fit of quasi-Poission and negative binomial models in R?
0
votes
1answer
184 views

Testing the significance of a main effect using model comparison, when an interaction is included

I am fitting a linear mixed effect model (using lmer), and want to obtain p-values via model comparison. As far as I understand it, the procedure is: (i) create a ...
0
votes
0answers
5 views

Comparing quality of completely unrelated models on completely unrelated data

Is there a good way to compare completely unrelated models at the top level, in order to be able to make sweeping statements such as "we can model number of clicks on an advertisement better than we ...
0
votes
0answers
18 views

DIC/MDIC is negative. Is this a red-flag or normal?

I have fit a few mixture models of multivariate normals. When calculating the MDIC (modified DIC) to compare the models, they come up as negative. In contrast, all examples in my lecture notes are ...
2
votes
1answer
45 views

Specifying multilevel model structure when random effects exhaust the population

I have been working with a dataset featuring observations at the county level for about 1300 of the ~3100 or so counties of the United States. These 1300 counties are drawn from every state in the ...
0
votes
1answer
17 views

Why compare measures of dispersion with ratios?

I'm writing a paper on a novel statistical model estimated using MCMC and am currently evaluating it using simulated data. We are comparing the performance of our model to an established model as ...
0
votes
0answers
16 views

Model comparison for random effects when items belong to multiple groups

This may be basic, but I'm not sure how to find an answer. I'm fitting a model on a time series with 1000 data points. Most of these points belong to more than one genre - adding the genre parameter ...
2
votes
1answer
71 views

Why are the trivial points included when calculating AUC?

I'm aware of some of the issues associated with using AUC for model comparison (see for example the articles referenced on Wikipedia: here, here, or here). But so far I have found nothing on an issue ...
2
votes
1answer
102 views

How to compare model fit of OLS and Poisson regression?

I have built two regression models to predict sales of different products based on a number of explanatory variables, with an offset term for the number of days each product was on sale. One is a ...
3
votes
1answer
276 views

Can one give an example(s) of when non-nested AIC model comparison is not useful for model selection?

Note: The question here is not the same as this one. Indeed, as an answer to that question the answer below was closed as unrelated, together with the suggestion (credit @gung) to ask a separate ...
1
vote
1answer
294 views

How do we compare different models performance [closed]

I am doing some research work regarding deep learning, I have a model and now I want to know and compare how my model performs as compared to others existing model.(like capsule networks, CNN ets..) ...
1
vote
2answers
72 views

different models? [closed]

VAR ARIMA ARDL, and GARCH What will each model tell me?
2
votes
2answers
42 views

Different metrics for training and evaluation/comparison?

When does one train and cross-validate some models on a particularly objective (say, likelihood), and then evaluate or compare the models based on a different metric (say, AUC)? Any advantages and ...
1
vote
0answers
34 views

Describing relative performance of models on the basis on log-loss / cross-entropy

Suppose I train classification algorithms on a given dataset with three model specifications: A, B, and C. On validation data, model A has an average log-loss of 1.0, model B has a log-loss of 0.75, ...
2
votes
0answers
212 views

Accuracy metric for comparing Time Series models?

I'm writing a blog post on forecasting time series with autoregression. In it, I compare the performance of SLR, ARIMA, and SARIMAX on forecasting the number of Home Sales in Seattle (see below). All ...