Skip to main content

Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

Filter by
Sorted by
Tagged with
1 vote
0 answers
19 views

GAMLSS - How to adjust AIC for log-transformation of variables [duplicate]

I have one GAMLSS model in which the independent and dependent variable are on the natural scale, and a second model in which both variables have been log-transformed. How do I adjust the AIC of the ...
0 votes
0 answers
30 views

Comparison of two different models with different variables

How to compare two models at two levels? For example, I want to model crime events at two levels (streets and census tracts). The models are logistic regression for the streets (occurrence of crime) ...
3 votes
1 answer
260 views

Model comparisons between linear models and linear mixed effects models

I'm interested to know if there is a method for comparing the performance of two different models, such as a linear model and a linear mixed effect model on the same data set? I'm interested to ...
1 vote
1 answer
244 views

Which significance test should be used to compare models based on Monte Carlo (holdout) cross-validation?

I have created two machine learning models and want to run significance tests on the results of various metrics (sensitivity, specificity, Cohen's kappa etc.) to see if there's any statistically ...
0 votes
0 answers
7 views

Interpretation Comparison Bayesian Models with Leave-One-Out Cross Validation

I'm currently working on comparing two Bayesian linear models using the brms package in R, with a dataset of 400 participants. The models differ in that one ...
1 vote
0 answers
56 views

Is comparing the AIC of a Bayesian and a frequentist model right?

I’m trying to fit a general linear model where the dependant variable is a probability. It is zero-inflated and continuous, then following the advice here blog of Ben Bolker, I separated my data pool ...
1 vote
0 answers
24 views

Comparing Adjusted $R^2$ Between Totally Different Models

I'm curious as to what extent adjusted $R^2$ can be used to compare models. If I had two different data sets and completely different models for both data sets, could I say something like the the ...
0 votes
0 answers
22 views

Computing Bayesian model averaged posteriors

The Bayesian model averaged posterior predictive distribution for new data $\tilde{y}$ given training data $y$, across a set of $M$ models $\mathcal{D} = \{D_{1}, ..., D_{M}\}$, is defined as: \begin{...
0 votes
0 answers
36 views

Compatibility of two atom spectra

I've measured in laboratory the wavelength of the emission lines of an unknown element. I now have these values in a set (called x) along with their relative errors. Now I need to compare my ...
0 votes
1 answer
3k views

Comparing models with different number of predictors

Given that the overall F-test of a multiple regression model has an F distribution, which depends on the number of predictors in the model, I understand why you cannot compare the F-statistics from ...
0 votes
0 answers
17 views

Linear mixed models with and without interactions

Here is the context: I asked two types of questions 8 times (8 "sessions" with 2 questions, each question has a "number" i.e. whether it is the first or second question asked, and ...
1 vote
1 answer
262 views

Comparing multiple models with something like a Diebold-Mariano test

I run many models on different samples (i.e., datasets which are slightly different from each other). I calculate the MSE for each model on these datasets. Now I want to compare these models to see if ...
2 votes
1 answer
1k views

Comparing the marginal effects of glm output to polr output

I have a dependent variable that is technically ordinal, so I ran a ordered probit model (polr). However, an ordered probit model does not produce any residuals ...
2 votes
1 answer
293 views

Significance test for comparing different 10-fold cross-validated Machine Learning Regressions

Is there a recommended significance test for comparing different 10-fold cross validated regressions? For instance, I want to compare the performance of LASSO against Random Forest for my dataset. ...
2 votes
1 answer
35 views

Significantly different pseudo R2 values

In a project I'm participating in, the Nagelkerke R2 value from a logistic regression model is used to compare the performance of a set of scores in explaining some binary data. The regression is ...
3 votes
2 answers
795 views

Which statistical test to compare same model with different parameters?

I have two datasets on people buying apples based on weight and price. One dataset in 2019 the other in 2020. I estimate a logit model with Utility = betaWeight * weight + betaPrice * price. Training ...
3 votes
1 answer
46 views

Is it ok to restrict data to uncensored patient in order to compare models

I have read this answer already "removing-censored-observation". I understand that removing censored data induce a bias in the analysis. But I have somehow a specific case. I am just ...
3 votes
3 answers
271 views

Comparing models with main effects and interactions

I have two models: Model 1: Only contains independent variable $x$, while $x$ is non-significant. Model 2: Contains $x$, $m$, and $x * m$, and $x * m$ is significant. How could I illustrate this ...
0 votes
0 answers
45 views

Using McNemar’s test to compare binary classifiers

I am evaluating multiple ML models which are performing the same binary classification task on the same dataset with the same train-val-test split. I am now looking to compare these models and since ...
1 vote
1 answer
2k views

An explanation of swap set analysis and swap set table

I'm trying to understand swap set analysis and the way results from such an analysis are presented. This analysis is a way to compare two models to determine which one is better. The results are ...
2 votes
1 answer
448 views

Comparability of classifier probabiliy estimates

Consider that you have 3 classification models ($model_1$, $model_2$ and $model_3$) that are designed to model whether or not customers are interested in different products of your company (binary ...
3 votes
1 answer
34 views

What method should be used to compare fit of different GAMs?

I am building GAMs of the following form (using R's mgcv notation) and would like to compare the fit between each model. ...
34 votes
3 answers
24k views

Prerequisites for AIC model comparison

What are exactly the prerequisites, that need to be fulfilled for AIC model comparison to work? I just came around this question when I did comparison like this: ...
2 votes
2 answers
707 views

What does it mean to measure model accuracy improvement?

Let's say I have some data, and a model X that models the data. Let's say that model X has a RMSE value of 1500 and a MAE value of 200. I build a new model Y, which is better than model X. Model Y ...
2 votes
1 answer
172 views

How to determine the best fitted model by AIC between lm(y~x),lm(log(y)~x), drc(y~x) in R

...
2 votes
1 answer
39 views

Diebold-Mariano test for one-step forecasts using Mean Absolute Scaled Error

This is my first time doing time series forecasting, so I am sorry for any inconsistencies in my question. But I have two different models that I want to compare. On Wikipedia, I read about Mean ...
2 votes
1 answer
64 views

P-Hacking concern: How valid is it to take statistically significant linear models from two different datasets and compare their coefficients?

I was wondering what if the following process can be considered p value hacking or just bad statistical practice. What I currently have are two subsets of one large parent dataset. These subsets are ...
1 vote
1 answer
430 views

SEM model comparison

I have problem choosing a better measurement model between two (i.e., A and B). Both models have acceptable fit indices. For example, the RMSEA of A is .056; the RMSEA of B is .067. By RMSEA, A seems ...
5 votes
1 answer
2k views

How to test significance in shift of a variable taken an other variable into the model (suest) in R?

I want to test in R whether an additional variable $x_2$ taken into a model influences an other variable significantly. (Note: This a revised repost of one in Stack Overflow where there was no ...
1 vote
0 answers
17 views

Metrics for comparring multiple models

I am building logistic regression models of some data to test a hypothesis and using cross-validation for each one. The cross-valdiation takes place within different subjects (mice in this case) of ...
0 votes
1 answer
39 views

Model comparison: with raw or normalized data?

I have developed a index of drug addiction risk whose formula is Index = 1/log10(a_given_variable). The raw values of the calculated ...
1 vote
0 answers
19 views

Comparison between theory and experiment using Mean Square Deviation

I want to compare the predictions of two theories A and B to some experimental data. The theories predict some quantity $y$ as a function of some other quantity $x$ by some analytical formula $y_A(x)$ ...
9 votes
1 answer
83 views

Testing a multifactor asset pricing model against another one

I have a sample of $(Y_{i,t},X_{1,i,t},X_{2,i,t})$ for $i=1,\dots,N$ and $t=1,\dots,T$. I want to figure out which data generating process (DGP) it comes from, DGP1 or DGP2. DGP1: $$ Y_{i,t}=\lambda_0+...
0 votes
0 answers
24 views

Which test is appropriate when testing between nested groups

Imagine that we want to analyze the outcome of a variable $Y$. The variable $Y$ represents the number of test passed, going from 1 to 7. I want to know if $Y$ varies among different groups, for ...
0 votes
0 answers
20 views

Interactions in regression and choosing appropriate model?

I’m running the following regression model, where MR is a continuous predictor and grade, gender, and race are all categorical. DV~MR + grade + gender + race I’m also interested in how the 3 ...
2 votes
1 answer
2k views

Equivalent measure to Matthews correlation coefficient, MCC, for multiclass classification

Thanks in advance for the help. MCC gives a measure of the quality of a binary classifier. I'm looking for a similar measure that can be used for a multi-class classifier. Ultimately what I would ...
0 votes
0 answers
144 views

Comparing the output distribution of two ML models

Consider a regression task (e.g. predicting house prices) with a given train and test sets. We start with constructing a linear regression model, in which we assume $y_i=X^T\beta+\epsilon$ with $E[\...
4 votes
1 answer
29 views

If there are several fixed effects of interest, and I want to test their inclusion via LRTs, in what order should I test them?

I have various potential fixed effect predictors for a linear mixed effect model. Some are control variables and some are predictors of interest (on their own and interactions). I am interested in ...
1 vote
6 answers
153 views

Regression model with good p-values but bad error

Suppose there is some data and all assumptions for regression are met. I fit 2 regression models now: Case 1: Regression model where all coefficients have significant $p$ values but overall the model ...
2 votes
0 answers
69 views

Why don't deep learning-based computer vision studies utilize statistical tests for comparisons? [closed]

Many readers familiar with scientific articles in the field of deep learning-based computer vision might have observed a common practice: the absence of statistical significance tests for comparing ...
0 votes
1 answer
44 views

BIC model comparison [closed]

My go/nogo task was recorded in 4 blocks, each with a new set of cue images, with the learning and therefore the Q-values being reset at the beginning of each block. I have fitted different versions ...
1 vote
1 answer
404 views

why are models compared over multiple datasets?

I cannot clearly understand why models are compared across multiple datasets. What practical problems do they aim to solve? Especially, in several papers, models are compared using datasets that are ...
3 votes
1 answer
121 views

Compare non-linear model parameter estimates between two groups with random effects

I have a treatment (2 groups) for which I want to compare parameter estimate of a sigmoid curve (i.e., do any of the three parameters differ significantly between my two treatment group). I am ...
0 votes
0 answers
27 views

How is it possible to compare powers of normality tests?

The famous paper [1] compares the power of several normality tests. Regardless of its result, my immediate question is, "How is that even possible?" Well, the definition of power in this ...
0 votes
0 answers
86 views

Comparison of regression coefficients, same y but two different X versions

I've seen versions of this question but not specifically answering the question of a statistical test to compare the beta coefficients for two independent variables (x1, x2) but the same dependent ...
0 votes
0 answers
24 views

De Longs test on ROC curves?

I have two different ROC curves for Model A and B. I wanted to calculate the DeLongs test to identify the statisical significance of the difference between the model. model_A_tpr = [0, 0.2, 0.4, 0.6, ...
6 votes
2 answers
194 views

How to compare fitted survival model with covariates vs. Kaplan-Meier?

How would one ideally compare fitted survival models with covariates vs. a Kaplan-Meier with a goal of getting an idea of whether the survival model describes the data well. E.g. in an example like ...
1 vote
0 answers
19 views

Predictive Diagnostic, Comparison of simulated data with observed data

The question is quite abstract, so I display it with only the essential information. Suppose that we have three models $B_{1}, B_{2}$ and $F_{3}$. The $B_{1}, B_{2}$ are Bayesian models and the $F_{3}$...
4 votes
0 answers
169 views

Trying to understand the theory behind my similar / better results than XGBoost using a calibrated linear model (GAM)

I just opened a discussion on reddit asking about why/how the calibrated linear models I've been training have been getting similar / better results than XGBoost in my experiments. I was told to cross ...
1 vote
0 answers
66 views

Comparing count groups/histograms with different bins

I'm looking at clustering techniques for which I have a ground truth. From the ground truth I can extract the cardinality (size) of clusters and I would like to test how well different methods can ...

1
2 3 4 5
12