Questions tagged [model-comparison]
Comparing two or more models fit to a common data set. It can be part of the process of "model selection".
614
questions
0
votes
0
answers
7
views
Comparing nested models for fixed main effect when there are interactions
I have a linear mixed model, which uses a multiply imputed dataset. I saw that LRT could be used to assess Fixed effect significance in linear mixed model. I used ...
1
vote
0
answers
12
views
How to interpret model fit in posterior predictive checks between two models that both capture the observations in its 1sigma?
I have two models aimed at explaining a single observed measurement $x_{obs}$:
Simple Model with 26 parameters $f_1(\theta)$.
Complex Model with 31 parameters $f_2(\theta)$.
Both models are assumed ...
0
votes
0
answers
8
views
How to Compare Binary Classifiers with Similar but Different Labels
I am interested in understanding the best way to compare two binary classifiers with similar but different labeling strategies. Here is the synopsis:
I am building a classifier to predict when a ...
- 969
0
votes
0
answers
7
views
Do MCMC chain lengths affect model comparison?
Say I have two Bayesian regression models fit to the the same data. For whatever reason, the second model takes longer to converge, so I run the MCMC sampling chain twice as long for the second model ...
- 343
2
votes
0
answers
51
views
How to perform data-driven choice of sample size
Say I want to compare the result of 2 regression model. One is the reference model and the second one is the new model, which I want to know if it performs better. Each model is evaluated with the ...
- 171
1
vote
0
answers
25
views
Comparing GLMM with LMM with -2*log-likelihood
Is it possible/recommended to compare the -2*Log-Likelihood (-2LL) value of a Generalized Linear Mixed Model (GLMM) against the -2LL value (and/or AIC/AICC/BIC) of a Linear Mixed Model (LMM) with the ...
- 135
6
votes
1
answer
217
views
Model comparison for nested regression models that are not symbolically nested
Say I have two nested models and I want to compare them, but they are parameterized differently so that no simple constraint (i.e., setting coefficients to 0) on the larger model corresponds exactly ...
- 35.7k
0
votes
0
answers
34
views
Bayesian: Formally comparing prior and posterior distributions
Consider Bayesian inference in the regression model:
$$ \begin{align*} y &= \beta_0+\beta_1x_1+\beta_2x_2 + \varepsilon \\ \varepsilon &\sim
\mathcal{N}(0,\sigma^2) \end{align*}$$
Suppose we ...
- 508
0
votes
0
answers
25
views
Confused about the results of a mixed effect model
I have a dataset of two continuous muscle activity levels A and B. I hypothesized that B is linearly correlated to A which is confirmed by a simple correlation test. And I have two categorical factors:...
- 1
4
votes
2
answers
190
views
Compare estimates between non-nested mixed-effects models
I have two mixed-effect linear regression models, both fit on the same same data and same outcome, but with slightly different predictors. Is there a way to compare (i.e., generate a p-value for the ...
- 1,794
0
votes
0
answers
38
views
Comparing Models with Unequal Sample Sizes
I have performed an association analysis where I have associatiated several different perdictor variables to a dependent variable. For each predictor, I run two models and compare them via the ...
- 75
0
votes
1
answer
33
views
ML V REML for Hypothesis Testing
I am curious about doing hypothesis testing with REML values via a likelihood ratio test, or whatever test would make sense for REML values, but these two highly upvoted answers to existing cross ...
- 425
2
votes
2
answers
201
views
How to compare two PMF generating functions?
I have two functions $f$ and $g$ that map an input vector $x_i$ to a probability mass function.
For example:
$$
x_i = [0,1,2] \\\\
f(x_i) \rightarrow [0.1, 0.3, 0.2, 0.2, 0.2] \\\\
g(x_i) \...
- 21
1
vote
0
answers
26
views
Calculate weight for GLM-quasi poisson model
I am running several models with the quasi-Poisson family. I am looking at data from vulture restaurants. Vulture count was modelled at each site as a function of either a linear or quadratic effect ...
- 11
0
votes
0
answers
42
views
Likelihood ratios not distributed as a chi2 distribution with the correct dof (Wilks' theorem)
I perform Bayesian inference on a mixture model such that $\mu$ is the mixture weight for a feature in the mixture
$p(x | \mu, \theta) = \mu p_{f}(x|\theta) + (1-\mu)p_{nf}(x|\theta)$
I have prior $p(\...
4
votes
1
answer
67
views
Measurement invariance and latent mean comparison between 3 groups
I performed an Exploratory Factor Analysis (EFA) followed by a Confirmatory Factor Analysis (CFA) in R, successfully assessing the model fit using indices such as TLI, CFI, RMSEA for a two-factor ...
- 43
0
votes
0
answers
32
views
Testing difference between two models using WAIC and degrees of freedom of WAIC
I am conducting Bayesian penalised regression, horseshoe specifically, in R using the bayesreg package see here. One model is nested within the other, i.e. to the second model I have simply used all ...
- 2,063
1
vote
0
answers
21
views
GAMLSS - How to adjust AIC for log-transformation of variables [duplicate]
I have one GAMLSS model in which the independent and dependent variable are on the natural scale, and a second model in which both variables have been log-transformed. How do I adjust the AIC of the ...
- 333
0
votes
0
answers
34
views
Comparison of two different models with different variables
How to compare two models at two levels? For example, I want to model crime events at two levels (streets and census tracts). The models are logistic regression for the streets (occurrence of crime) ...
- 43
0
votes
0
answers
14
views
Interpretation Comparison Bayesian Models with Leave-One-Out Cross Validation
I'm currently working on comparing two Bayesian linear models using the brms package in R, with a dataset of 400 participants. The models differ in that one ...
- 21
1
vote
0
answers
27
views
Comparing Adjusted $R^2$ Between Totally Different Models
I'm curious as to what extent adjusted $R^2$ can be used to compare models. If I had two different data sets and completely different models for both data sets, could I say something like the the ...
0
votes
0
answers
24
views
Computing Bayesian model averaged posteriors
The Bayesian model averaged posterior predictive distribution for new data $\tilde{y}$ given training data $y$, across a set of $M$ models $\mathcal{D} = \{D_{1}, ..., D_{M}\}$, is defined as:
\begin{...
- 185
0
votes
0
answers
19
views
Linear mixed models with and without interactions
Here is the context: I asked two types of questions 8 times (8 "sessions" with 2 questions, each question has a "number" i.e. whether it is the first or second question asked, and ...
- 15
0
votes
0
answers
40
views
Compatibility of two atom spectra
I've measured in laboratory the wavelength of the emission lines of an unknown element.
I now have these values in a set (called x) along with their relative errors.
Now I need to compare my ...
- 1
1
vote
0
answers
62
views
Is comparing the AIC of a Bayesian and a frequentist model right?
I’m trying to fit a general linear model where the dependant variable is a probability. It is zero-inflated and continuous, then following the advice here blog of Ben Bolker, I separated my data pool ...
2
votes
1
answer
39
views
Significantly different pseudo R2 values
In a project I'm participating in, the Nagelkerke R2 value from a logistic regression model is used to compare the performance of a set of scores in explaining some binary data. The regression is ...
- 21
3
votes
1
answer
51
views
Is it ok to restrict data to uncensored patient in order to compare models
I have read this answer already "removing-censored-observation". I understand that removing censored data induce a bias in the analysis. But I have somehow a specific case.
I am just ...
1
vote
0
answers
67
views
Using McNemar’s test to compare binary classifiers
I am evaluating multiple ML models which are performing the same binary classification task on the same dataset with the same train-val-test split. I am now looking to compare these models and since ...
3
votes
1
answer
58
views
What method should be used to compare fit of different GAMs?
I am building GAMs of the following form (using R's mgcv notation) and would like to compare the fit between each model.
...
- 585
2
votes
1
answer
175
views
2
votes
1
answer
64
views
Diebold-Mariano test for one-step forecasts using Mean Absolute Scaled Error
This is my first time doing time series forecasting, so I am sorry for any inconsistencies in my question. But I have two different models that I want to compare. On Wikipedia, I read about Mean ...
- 23
2
votes
1
answer
77
views
P-Hacking concern: How valid is it to take statistically significant linear models from two different datasets and compare their coefficients?
I was wondering what if the following process can be considered p value hacking or just bad statistical practice. What I currently have are two subsets of one large parent dataset. These subsets are ...
- 115
1
vote
0
answers
17
views
Metrics for comparring multiple models
I am building logistic regression models of some data to test a hypothesis and using cross-validation for each one. The cross-valdiation takes place within different subjects (mice in this case) of ...
- 468
0
votes
1
answer
48
views
Model comparison: with raw or normalized data?
I have developed a index of drug addiction risk whose formula is Index = 1/log10(a_given_variable). The raw values of the calculated ...
- 1
3
votes
3
answers
402
views
Comparing models with main effects and interactions
I have two models:
Model 1: Only contains independent variable $x$, while $x$ is non-significant.
Model 2: Contains $x$, $m$, and $x * m$, and $x * m$ is significant.
How could I illustrate this ...
- 31
0
votes
0
answers
36
views
Which test is appropriate when testing between nested groups
Imagine that we want to analyze the outcome of a variable $Y$. The variable $Y$ represents the number of test passed, going from 1 to 7.
I want to know if $Y$ varies among different groups, for ...
- 472
8
votes
1
answer
85
views
Testing a multifactor asset pricing model against another one
I have a sample of $(Y_{i,t},X_{1,i,t},X_{2,i,t})$ for $i=1,\dots,N$ and $t=1,\dots,T$. I want to figure out which data generating process (DGP) it comes from, DGP1 or DGP2.
DGP1:
$$
Y_{i,t}=\lambda_0+...
- 68.8k
0
votes
0
answers
21
views
Interactions in regression and choosing appropriate model?
I’m running the following regression model, where MR is a continuous predictor and grade, gender, and race are all categorical.
DV~MR + grade + gender + race
I’m also interested in how the 3 ...
- 11
0
votes
0
answers
158
views
Comparing the output distribution of two ML models
Consider a regression task (e.g. predicting house prices) with a given train and test sets.
We start with constructing a linear regression model, in which we assume $y_i=X^T\beta+\epsilon$ with $E[\...
- 4,017
1
vote
6
answers
197
views
Regression model with good p-values but bad error
Suppose there is some data and all assumptions for regression are met. I fit 2 regression models now:
Case 1: Regression model where all coefficients have significant $p$ values but overall the model ...
2
votes
0
answers
89
views
Why don't deep learning-based computer vision studies utilize statistical tests for comparisons? [closed]
Many readers familiar with scientific articles in the field of deep learning-based computer vision might have observed a common practice: the absence of statistical significance tests for comparing ...
- 53
0
votes
1
answer
51
views
BIC model comparison [closed]
My go/nogo task was recorded in 4 blocks, each with a new set of cue images, with the learning and therefore the Q-values being reset at the beginning of each block.
I have fitted different versions ...
- 3
1
vote
1
answer
418
views
why are models compared over multiple datasets?
I cannot clearly understand why models are compared across multiple datasets. What practical problems do they aim to solve? Especially, in several papers, models are compared using datasets that are ...
- 13
3
votes
1
answer
149
views
Compare non-linear model parameter estimates between two groups with random effects
I have a treatment (2 groups) for which I want to compare parameter estimate of a sigmoid curve (i.e., do any of the three parameters differ significantly between my two treatment group).
I am ...
- 304
0
votes
0
answers
28
views
How is it possible to compare powers of normality tests?
The famous paper [1] compares the power of several normality
tests. Regardless of its result, my immediate question is, "How
is that even possible?"
Well, the definition of power in this ...
- 235
0
votes
0
answers
100
views
Comparison of regression coefficients, same y but two different X versions
I've seen versions of this question but not specifically answering the question of a statistical test to compare the beta coefficients for two independent variables (x1, x2) but the same dependent ...
- 11
0
votes
0
answers
26
views
De Longs test on ROC curves?
I have two different ROC curves for Model A and B. I wanted to calculate the DeLongs test to identify the statisical significance of the difference between the model.
model_A_tpr = [0, 0.2, 0.4, 0.6, ...
- 101
6
votes
2
answers
207
views
How to compare fitted survival model with covariates vs. Kaplan-Meier?
How would one ideally compare fitted survival models with covariates vs. a Kaplan-Meier with a goal of getting an idea of whether the survival model describes the data well.
E.g. in an example like ...
- 34.2k
1
vote
0
answers
19
views
Predictive Diagnostic, Comparison of simulated data with observed data
The question is quite abstract, so I display it with only the essential information.
Suppose that we have three models $B_{1}, B_{2}$ and $F_{3}$. The $B_{1}, B_{2}$ are Bayesian models and the $F_{3}$...
- 2,286
4
votes
0
answers
179
views
Trying to understand the theory behind my similar / better results than XGBoost using a calibrated linear model (GAM)
I just opened a discussion on reddit asking about why/how the calibrated linear models I've been training have been getting similar / better results than XGBoost in my experiments. I was told to cross ...
- 41