Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

1
vote
0answers
8 views

Which is the more reliable method for reporting classification results in deep learning?

I have two methods to compare, the one which reports weighted F1-score with imbalanced data and, the one which reports better F1-score with balanced data. I am confused as to which method's ...
2
votes
1answer
31 views

Compare two mixed-effect models differing on the nested effect

I need to compare two models with repeated measures (I'm using using lme4 or glmmTMB for the random effects) ...
0
votes
1answer
25 views

Interpreting ANOVA f-test for model comparison

In the book ISLR, there's a lab activity which compares the nested models. ...
7
votes
2answers
190 views

Logistic regression BIC: what's the right N?

TL;DR: Which $N$ is correct for BIC in logistic regression, the aggregated binomial or Bernoulli $N$? UPDATES AT BOTTOM Suppose I have a data set to which I'd like to apply logistic regression. For ...
0
votes
0answers
17 views

BIC score for Gaussian Mixture Model

I'm not sure how to compute a BIC score for multiple classes. For exmaple , I have a supervised problem with 3 classes.I fit 3 gaussian using MLE. Then, if I want to compute BIC score: I have to ...
0
votes
0answers
16 views

Metrics for comparing a multi-class model vs. a multi-label model?

The dataset is a multi-label dataset, where each item can have more than one labels. I first trained a multi-class classifier by randomly select the label for each item at each iteration, and ...
1
vote
0answers
48 views

How do you / can you compare Bayesian vs. frequentist regression models?

I am working on a regression model to predict a target variable in a dataset with over 100 features. Three different regression models are defined and fit in order to compare their performance using $...
1
vote
0answers
23 views

Model comparison in R

I want to stack three models (RF, SVM, ANN) using Stochastic Gradient Boosting as the meta-learner. After computing predictions from my sub-models, I used the ModelCor function in R to compute a model ...
0
votes
0answers
23 views

Kolmogorov Smirnov, CVM, AD all reject Null hypothesis - comparing two samples

I have an empirical sample (with N >10000) and I fit it to a beta-distribution. Now, as a goodness of fit, I first looked at the density function plots of the empirical and the estimated beta ...
8
votes
2answers
156 views

How can I compare models without fitting?

Regression and machine learning are used in the natural sciences to test hypotheses, estimate parameters, and make predictions by fitting models to data. However, when I have an a priori model, I don'...
0
votes
0answers
20 views

compare the outputs of two different ML models

Let's imagine I have two completely different multi-class ML models, let's call them ML1 and ML2. The models were trained on completely different data with different target classes. As an output, the ...
1
vote
0answers
26 views

Using unnormalized log likelihood for model comparison with different features?

Is it possible to directly use the log likelihood from fitting a model, for the purpose of model comparison? For example, if I'm using a logistic regression model, and I want to see if adding ...
1
vote
0answers
111 views

How to compare non-nested models (with different number of items) in CFA?

I investigate the factor structur of an existing scale in a data set with N = 714 (using lavaan, R). In this context I would like to compare to alternative models: Model 1: Two factors with 5 items ...
1
vote
1answer
51 views

Bootstrap Hypothesis Test: Comparing the Performance of two Models

I have two models $A$ and $B$ as well as a training set $X$. I want to test whether there is a significant difference in performance between model $A$ and $B$. I'm attempting to do this via ...
3
votes
0answers
71 views

Is the use of loglik or AIC to compare logit/probit/cloglog models valid?

I would like to know whether I can use AIC, or if the models have the same number of predictors, the log-likelihood, to compare logit vs probit vs cloglog models (fitted for instance with glmer or ...
0
votes
0answers
16 views

Comparing multiple hurdle to compensatory scoring (of tests)

I plan to do both Monte Carlo and real world research to compare the efficacy of three approaches to combining test scores: (1) fully compensatory scoring, (2) multiple hurdle, and (3) a combination ...
0
votes
1answer
30 views

Predicting proportion of individuals belonging to different classes

I am trying to compare performances of different classifiers to predict some data. The variable I wand to predict is binary (A or B), and I have a bunch of predictive variables to predict to what ...
0
votes
0answers
8 views

Comparing two Models gained from process discovery

I have a rather basic question on process mining that I however hardly get answered: currently, I do some research on regulated learning in educational/psychological research: I already drew up two ...
0
votes
0answers
43 views

Model comparison when using Harrell's bootstrap-based optimism-correction Internal Validation procedure

I have 2 non-nested models that I would like to compare with respect to their predictive accuracy (concordance index). I have used Harrell's (Efron-Gong) bootstrap-based optimism-correction internal ...
2
votes
1answer
83 views

summary of lm to compare models instead of ANOVA

What does it mean to use anova to compare models? For example, if I have two models $$ \text{model}_1 \colon y = b_0 + b_1 x_1 + b_2 x_2 $$ and $$ \text{...
1
vote
0answers
14 views

Model comparison for predictors with differently shaped distributions

I want to use model comparison to evaluate the claim "Predictor A outperforms Predictor B in modeling Dependent X". The (potential) problem is that within my sample, Predictor A and Predictor B have ...
0
votes
0answers
8 views

Checking equality of coefficients from two different count data regression models

Suppose, I have cluster survey data. Individuals are nested within cluster. My outcome variable of interests are two count variables Y1 and Y2. I have a set of predictor variables X1, X2, ... X5. Now ...
1
vote
1answer
21 views

Compare a real-world process with model prediction

I have a real-world time series process with an dependent variable Y responds to multiple independent variables (not necessary a linear response but can assume so if necessary). A non-statistical ...
0
votes
0answers
51 views

Compairing the fit of quasi-Poisson and negative binomial models

Is there any way to compare the fit of quasi-Poission and negative binomial models in R?
0
votes
1answer
108 views

Testing the significance of a main effect using model comparison, when an interaction is included

I am fitting a linear mixed effect model (using lmer), and want to obtain p-values via model comparison. As far as I understand it, the procedure is: (i) create a ...
0
votes
0answers
5 views

Comparing quality of completely unrelated models on completely unrelated data

Is there a good way to compare completely unrelated models at the top level, in order to be able to make sweeping statements such as "we can model number of clicks on an advertisement better than we ...
0
votes
0answers
12 views

DIC/MDIC is negative. Is this a red-flag or normal?

I have fit a few mixture models of multivariate normals. When calculating the MDIC (modified DIC) to compare the models, they come up as negative. In contrast, all examples in my lecture notes are ...
2
votes
1answer
43 views

Specifying multilevel model structure when random effects exhaust the population

I have been working with a dataset featuring observations at the county level for about 1300 of the ~3100 or so counties of the United States. These 1300 counties are drawn from every state in the ...
0
votes
1answer
15 views

Why compare measures of dispersion with ratios?

I'm writing a paper on a novel statistical model estimated using MCMC and am currently evaluating it using simulated data. We are comparing the performance of our model to an established model as ...
0
votes
0answers
16 views

Model comparison for random effects when items belong to multiple groups

This may be basic, but I'm not sure how to find an answer. I'm fitting a model on a time series with 1000 data points. Most of these points belong to more than one genre - adding the genre parameter ...
2
votes
1answer
68 views

Why are the trivial points included when calculating AUC?

I'm aware of some of the issues associated with using AUC for model comparison (see for example the articles referenced on Wikipedia: here, here, or here). But so far I have found nothing on an issue ...
3
votes
1answer
223 views

Can one give an example(s) of when non-nested AIC model comparison is not useful for model selection?

Note: The question here is not the same as this one. Indeed, as an answer to that question the answer below was closed as unrelated, together with the suggestion (credit @gung) to ask a separate ...
1
vote
1answer
261 views

How do we compare different models performance [closed]

I am doing some research work regarding deep learning, I have a model and now I want to know and compare how my model performs as compared to others existing model.(like capsule networks, CNN ets..) ...
1
vote
2answers
68 views

Econometric models? [closed]

I'm confused about econometrics models. Let's say I used a daily stock market index for 5 years, and I want to see how oil price affect the stock markets return , let's assume ( as a fact ) that oil ...
2
votes
2answers
41 views

Different metrics for training and evaluation/comparison?

When does one train and cross-validate some models on a particularly objective (say, likelihood), and then evaluate or compare the models based on a different metric (say, AUC)? Any advantages and ...
1
vote
0answers
27 views

Describing relative performance of models on the basis on log-loss / cross-entropy

Suppose I train classification algorithms on a given dataset with three model specifications: A, B, and C. On validation data, model A has an average log-loss of 1.0, model B has a log-loss of 0.75, ...
1
vote
0answers
146 views

Accuracy metric for comparing Time Series models?

I'm writing a blog post on forecasting time series with autoregression. In it, I compare the performance of SLR, ARIMA, and SARIMAX on forecasting the number of Home Sales in Seattle (see below). All ...
1
vote
0answers
67 views

Is this an appropriate way to compare simulated and measured data with small sample size, and are there alternatives?

I have a probably fairly basic question which I couldn't find an answer to on here or anywhere else. Would really appreciate any thoughts on this :) I am modelling the propagation of sound through a ...
0
votes
2answers
20 views

Stats test for nominal data:

I have to validate if two different methodologies yield statistically similar results to prove if we can use either of the methodologies. both the methodologies use the same sample, i.e same group of ...
1
vote
1answer
73 views

Comparing measures of association

In a survival analysis, I would like to prove that a variable Y is more associated with an outcome than another variable Y'. My ...
1
vote
0answers
70 views

Comparing log-transformed and non-log transformed models

Suppose I have the following two models fitted with constrained linear least squares: Model 1: $Y = \beta_1X_1 + \ldots + \beta_kX_k + \varepsilon$ Model 2: $\log_{10}(Y) = \beta_1\log_{10}(X_1) + \...
3
votes
1answer
106 views

Reporting betareg outcome - how to compare non-nested models?

I am looking for advice how to gain and report results using beta regression for an ANCOVA-like model. My model is as follows: ...
1
vote
0answers
32 views

Comparing predictiveness of alternative class definitions using linear regression

I am comparing highly-correlated ways to classify patients as sick (1) or well (0). I want to see whether the way you define sickness impacts how well sickness predicts clinical outcomes. I have ...
1
vote
0answers
10 views

How can Degrees of Freedom increase for Mixed Effects Model Comparison w/ Chi Square?

I'm using lme4 for mixed effects linear regression, and I'd like to compare two models using anova(). However, I'm perplexed by ...
2
votes
1answer
36 views

Test which one of two (mechanistic / non-regression) models fits the observations better

There's a large number of similar questions, but they mostly seem to be about regression models and deal with cross validation, comparison of nested statistical models via likelihood-ratio, etc. ...
0
votes
0answers
27 views

Comparing metrics with mean and standard deviation

I have been running some experiments for a problem using different models. Those experiment run on 10 different datasets and each of them yields a value for RMSE. After all the 10 experiments are ...
0
votes
0answers
41 views

How to test significance in shift of a variable taken an other variable into the model (suest) in R?

I want to test in R whether an additional variable $x_2$ taken into a model influences an other variable significantly. (Note: This a revised repost of one in Stack Overflow where there was no ...
0
votes
0answers
50 views

model comparison of mixed effects using Anova()

I want to use model comparison to estimate the significance the effects used in a linear mixed effects model (using R's lmer()). In theory, I could create multiple ...
7
votes
4answers
257 views

How to test if the process that generated a time-series has changed over time

Problem I have time-series data generated by a machine over two disjoint periods of time - roughly one month in 2016 and another month in 2018. It is hypothesized by domain experts that at each time ...
0
votes
0answers
19 views

How to compare beta coefficients estimated by negative binomial GLM?

I have run two negative binomial regression models on different sets of counts data, and have two beta coefficient estimates and their standard errors for a variable of interest. I would like to use ...