Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

1
vote
1answer
34 views

Bootstrap Hypothesis Test: Comparing the Performance of two Models

I have two models $A$ and $B$ as well as a training set $X$. I want to test whether there is a significant difference in performance between model $A$ and $B$. I'm attempting to do this via ...
3
votes
0answers
66 views

Is the use of loglik or AIC to compare logit/probit/cloglog models valid?

I would like to know whether I can use AIC, or if the models have the same number of predictors, the log-likelihood, to compare logit vs probit vs cloglog models (fitted for instance with glmer or ...
0
votes
0answers
10 views

Comparing multiple hurdle to compensatory scoring (of tests)

I plan to do both Monte Carlo and real world research to compare the efficacy of three approaches to combining test scores: (1) fully compensatory scoring, (2) multiple hurdle, and (3) a combination ...
0
votes
1answer
26 views

Predicting proportion of individuals belonging to different classes

I am trying to compare performances of different classifiers to predict some data. The variable I wand to predict is binary (A or B), and I have a bunch of predictive variables to predict to what ...
0
votes
0answers
8 views

Comparing two Models gained from process discovery

I have a rather basic question on process mining that I however hardly get answered: currently, I do some research on regulated learning in educational/psychological research: I already drew up two ...
0
votes
0answers
31 views

Model comparison when using Harrell's bootstrap-based optimism-correction Internal Validation procedure

I have 2 non-nested models that I would like to compare with respect to their predictive accuracy (concordance index). I have used Harrell's (Efron-Gong) bootstrap-based optimism-correction internal ...
2
votes
1answer
56 views

summary of lm to compare models instead of ANOVA

What does it mean to use anova to compare models? For example, if I have two models $$ \text{model}_1 \colon y = b_0 + b_1 x_1 + b_2 x_2 $$ and $$ \text{...
1
vote
0answers
14 views

Model comparison for predictors with differently shaped distributions

I want to use model comparison to evaluate the claim "Predictor A outperforms Predictor B in modeling Dependent X". The (potential) problem is that within my sample, Predictor A and Predictor B have ...
0
votes
0answers
7 views

Checking equality of coefficients from two different count data regression models

Suppose, I have cluster survey data. Individuals are nested within cluster. My outcome variable of interests are two count variables Y1 and Y2. I have a set of predictor variables X1, X2, ... X5. Now ...
1
vote
1answer
20 views

Compare a real-world process with model prediction

I have a real-world time series process with an dependent variable Y responds to multiple independent variables (not necessary a linear response but can assume so if necessary). A non-statistical ...
-1
votes
0answers
12 views

What is the proper way to model reading & response times

I plan to measure the time it takes users to read and respond to a series of similar multiple-choice questions. Before starting the experiment, I'd like to think of a proper distribution that can be ...
0
votes
1answer
48 views

Testing the significance of a main effect using model comparison, when an interaction is included

I am fitting a linear mixed effect model (using lmer), and want to obtain p-values via model comparison. As far as I understand it, the procedure is: (i) create a ...
-1
votes
0answers
14 views

What is the best way to compare the results of the proposed technique with existing if a different dataset is used?

I am working on the multi-disease classification problem and my trained algorithm can classify the input image into one of five non-overlapping classes. I want to compare my algorithm results with the ...
0
votes
0answers
4 views

Comparing quality of completely unrelated models on completely unrelated data

Is there a good way to compare completely unrelated models at the top level, in order to be able to make sweeping statements such as "we can model number of clicks on an advertisement better than we ...
0
votes
0answers
9 views

DIC/MDIC is negative. Is this a red-flag or normal?

I have fit a few mixture models of multivariate normals. When calculating the MDIC (modified DIC) to compare the models, they come up as negative. In contrast, all examples in my lecture notes are ...
2
votes
1answer
41 views

Specifying multilevel model structure when random effects exhaust the population

I have been working with a dataset featuring observations at the county level for about 1300 of the ~3100 or so counties of the United States. These 1300 counties are drawn from every state in the ...
0
votes
0answers
6 views

Compare the performance of a new model with the existing model

The challenge for me here is that I do not know the test/train datasets of the EXISTING model but I have the model with me (it is a logistic regression equation). I believe the NEW model's AUC on ...
0
votes
0answers
28 views

Compare binary classification models

I am comparing the predictive performance of two binary classification models. The first model output is discrete binary labels while the second model output is continuous scores. I am wondering what ...
0
votes
1answer
15 views

Why compare measures of dispersion with ratios?

I'm writing a paper on a novel statistical model estimated using MCMC and am currently evaluating it using simulated data. We are comparing the performance of our model to an established model as ...
0
votes
0answers
16 views

Model comparison for random effects when items belong to multiple groups

This may be basic, but I'm not sure how to find an answer. I'm fitting a model on a time series with 1000 data points. Most of these points belong to more than one genre - adding the genre parameter ...
2
votes
1answer
62 views

Why are the trivial points included when calculating AUC?

I'm aware of some of the issues associated with using AUC for model comparison (see for example the articles referenced on Wikipedia: here, here, or here). But so far I have found nothing on an issue ...
3
votes
1answer
189 views

Can one give an example(s) of when non-nested AIC model comparison is not useful for model selection?

Note: The question here is not the same as this one. Indeed, as an answer to that question the answer below was closed as unrelated, together with the suggestion (credit @gung) to ask a separate ...
1
vote
1answer
201 views

How do we compare different models performance [closed]

I am doing some research work regarding deep learning, I have a model and now I want to know and compare how my model performs as compared to others existing model.(like capsule networks, CNN ets..) ...
1
vote
2answers
67 views

Econometric models? [closed]

I'm confused about econometrics models. Let's say I used a daily stock market index for 5 years, and I want to see how oil price affect the stock markets return , let's assume ( as a fact ) that oil ...
2
votes
2answers
40 views

Different metrics for training and evaluation/comparison?

When does one train and cross-validate some models on a particularly objective (say, likelihood), and then evaluate or compare the models based on a different metric (say, AUC)? Any advantages and ...
1
vote
0answers
23 views

Describing relative performance of models on the basis on log-loss / cross-entropy

Suppose I train classification algorithms on a given dataset with three model specifications: A, B, and C. On validation data, model A has an average log-loss of 1.0, model B has a log-loss of 0.75, ...
1
vote
0answers
82 views

Accuracy metric for comparing Time Series models?

I'm writing a blog post on forecasting time series with autoregression. In it, I compare the performance of SLR, ARIMA, and SARIMAX on forecasting the number of Home Sales in Seattle (see below). All ...
1
vote
0answers
65 views

Is this an appropriate way to compare simulated and measured data with small sample size, and are there alternatives?

I have a probably fairly basic question which I couldn't find an answer to on here or anywhere else. Would really appreciate any thoughts on this :) I am modelling the propagation of sound through a ...
0
votes
2answers
17 views

Stats test for nominal data:

I have to validate if two different methodologies yield statistically similar results to prove if we can use either of the methodologies. both the methodologies use the same sample, i.e same group of ...
1
vote
1answer
71 views

Comparing measures of association

In a survival analysis, I would like to prove that a variable Y is more associated with an outcome than another variable Y'. My ...
1
vote
0answers
47 views

Comparing log-transformed and non-log transformed models

Suppose I have the following two models fitted with constrained linear least squares: Model 1: $Y = \beta_1X_1 + \ldots + \beta_kX_k + \varepsilon$ Model 2: $\log_{10}(Y) = \beta_1\log_{10}(X_1) + \...
3
votes
1answer
73 views

Reporting betareg outcome - how to compare non-nested models?

I am looking for advice how to gain and report results using beta regression for an ANCOVA-like model. My model is as follows: ...
1
vote
0answers
32 views

Comparing predictiveness of alternative class definitions using linear regression

I am comparing highly-correlated ways to classify patients as sick (1) or well (0). I want to see whether the way you define sickness impacts how well sickness predicts clinical outcomes. I have ...
1
vote
0answers
10 views

How can Degrees of Freedom increase for Mixed Effects Model Comparison w/ Chi Square?

I'm using lme4 for mixed effects linear regression, and I'd like to compare two models using anova(). However, I'm perplexed by ...
2
votes
1answer
35 views

Test which one of two (mechanistic / non-regression) models fits the observations better

There's a large number of similar questions, but they mostly seem to be about regression models and deal with cross validation, comparison of nested statistical models via likelihood-ratio, etc. ...
0
votes
0answers
26 views

Comparing metrics with mean and standard deviation

I have been running some experiments for a problem using different models. Those experiment run on 10 different datasets and each of them yields a value for RMSE. After all the 10 experiments are ...
0
votes
0answers
32 views

How to test significance in shift of a variable taken an other variable into the model (suest) in R?

I want to test in R whether an additional variable $x_2$ taken into a model influences an other variable significantly. (Note: This a revised repost of one in Stack Overflow where there was no ...
0
votes
0answers
34 views

model comparison of mixed effects using Anova()

I want to use model comparison to estimate the significance the effects used in a linear mixed effects model (using R's lmer()). In theory, I could create multiple ...
7
votes
4answers
236 views

How to test if the process that generated a time-series has changed over time

Problem I have time-series data generated by a machine over two disjoint periods of time - roughly one month in 2016 and another month in 2018. It is hypothesized by domain experts that at each time ...
0
votes
0answers
20 views

Regression model comparisons or sum coding?

It seems like the standard approach to assess main and interaction effects in regression is to do model comparisons between a full model and a simpler model without the effect(s) of interest using ...
0
votes
0answers
13 views

How to compare beta coefficients estimated by negative binomial GLM?

I have run two negative binomial regression models on different sets of counts data, and have two beta coefficient estimates and their standard errors for a variable of interest. I would like to use ...
1
vote
0answers
32 views

Is it possible to calculte the BIC in LASSO cases if p>n

I thinks the question should be clear. Is it possible to calculate the BIC criterion in cases where p>n with the "traditional" BIC Formula ? I know that there are some extensions of information ...
1
vote
0answers
30 views

Comparing hundreds of regression models in R efficiently [closed]

I'm working with a data set of housing prices in Seattle from Kaggle.com. I have identified 7 variables that are significant in a linear regression model for modeling/predicting the prices of the ...
0
votes
0answers
19 views

Compare Hazard Ratios from cross sectional data at two time periods

Can I fit hazard rate models on two different datasets in time (with same features and different observations) and compare the observed hazard ratios? For example, I would like to find if gender and ...
0
votes
0answers
20 views

The Best statistical test for comparing two models with the same data [duplicate]

I am not referring to the F test, which could be misleading. Ok, so I have one set of Data and two models to describe the data. Problem is, the second model has the same exact parameters of model 1, ...
1
vote
2answers
60 views

Compare models using residuals

Is it statistically correct to compare two models using their residuals? For example, I have two dose-response models, then am comparing their residuals and concluding they are not statistically ...
2
votes
0answers
83 views

Why is a variable with a comparatively smaller odds ratio statistically significant (when the larger OR is not)? [closed]

I have two predictors in a binary logistic regression model, where both variables are on the same scale (standardized). I find it puzzling that the variable with a smaller OR is significant at p <. ...
0
votes
0answers
8 views

Winbugs Criterion

Suppose that we have two general linear models $(M_{1})$ (an additive) and $(M_{2})$ (an multiplicative) .For each of them we use winbugs and calculate DIC in order to compare and find the best model. ...
1
vote
1answer
36 views

Is augmentation of treatment effect caused significantly by mediator?

I want to find out if the influence of potential mediator P on treatment effect D is significant. In the two models below $\beta_1$ and $\lambda_1$ are different and I don't know how to test if the ...
2
votes
0answers
89 views

Compare and quantify relative improvement in ROC AUC scores?

What is an appropriate method for comparing relative improvement in model performance across different problems? For example, say I have three different datasets/problems a, b, c, and two models for ...