Skip to main content

Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

Filter by
Sorted by
Tagged with
1 vote
0 answers
19 views

GAMLSS - How to adjust AIC for log-transformation of variables [duplicate]

I have one GAMLSS model in which the independent and dependent variable are on the natural scale, and a second model in which both variables have been log-transformed. How do I adjust the AIC of the ...
Peder Holman's user avatar
0 votes
0 answers
30 views

Comparison of two different models with different variables

How to compare two models at two levels? For example, I want to model crime events at two levels (streets and census tracts). The models are logistic regression for the streets (occurrence of crime) ...
AmirMohammad's user avatar
0 votes
0 answers
7 views

Interpretation Comparison Bayesian Models with Leave-One-Out Cross Validation

I'm currently working on comparing two Bayesian linear models using the brms package in R, with a dataset of 400 participants. The models differ in that one ...
JKas's user avatar
  • 21
1 vote
0 answers
24 views

Comparing Adjusted $R^2$ Between Totally Different Models

I'm curious as to what extent adjusted $R^2$ can be used to compare models. If I had two different data sets and completely different models for both data sets, could I say something like the the ...
Michael Jones's user avatar
0 votes
0 answers
22 views

Computing Bayesian model averaged posteriors

The Bayesian model averaged posterior predictive distribution for new data $\tilde{y}$ given training data $y$, across a set of $M$ models $\mathcal{D} = \{D_{1}, ..., D_{M}\}$, is defined as: \begin{...
user_15's user avatar
  • 185
0 votes
0 answers
17 views

Linear mixed models with and without interactions

Here is the context: I asked two types of questions 8 times (8 "sessions" with 2 questions, each question has a "number" i.e. whether it is the first or second question asked, and ...
statuser's user avatar
0 votes
0 answers
36 views

Compatibility of two atom spectra

I've measured in laboratory the wavelength of the emission lines of an unknown element. I now have these values in a set (called x) along with their relative errors. Now I need to compare my ...
yZemp's user avatar
  • 1
1 vote
0 answers
56 views

Is comparing the AIC of a Bayesian and a frequentist model right?

I’m trying to fit a general linear model where the dependant variable is a probability. It is zero-inflated and continuous, then following the advice here blog of Ben Bolker, I separated my data pool ...
Auvray alexandre's user avatar
2 votes
1 answer
35 views

Significantly different pseudo R2 values

In a project I'm participating in, the Nagelkerke R2 value from a logistic regression model is used to compare the performance of a set of scores in explaining some binary data. The regression is ...
Kristoffer's user avatar
3 votes
1 answer
46 views

Is it ok to restrict data to uncensored patient in order to compare models

I have read this answer already "removing-censored-observation". I understand that removing censored data induce a bias in the analysis. But I have somehow a specific case. I am just ...
Tamsina Ludwig's user avatar
0 votes
0 answers
45 views

Using McNemar’s test to compare binary classifiers

I am evaluating multiple ML models which are performing the same binary classification task on the same dataset with the same train-val-test split. I am now looking to compare these models and since ...
SmollPangolin's user avatar
3 votes
1 answer
33 views

What method should be used to compare fit of different GAMs?

I am building GAMs of the following form (using R's mgcv notation) and would like to compare the fit between each model. ...
geoscience123's user avatar
2 votes
1 answer
172 views

How to determine the best fitted model by AIC between lm(y~x),lm(log(y)~x), drc(y~x) in R

...
user408308's user avatar
2 votes
1 answer
37 views

Diebold-Mariano test for one-step forecasts using Mean Absolute Scaled Error

This is my first time doing time series forecasting, so I am sorry for any inconsistencies in my question. But I have two different models that I want to compare. On Wikipedia, I read about Mean ...
Oskar Weber's user avatar
2 votes
1 answer
64 views

P-Hacking concern: How valid is it to take statistically significant linear models from two different datasets and compare their coefficients?

I was wondering what if the following process can be considered p value hacking or just bad statistical practice. What I currently have are two subsets of one large parent dataset. These subsets are ...
Syuma's user avatar
  • 115
1 vote
0 answers
17 views

Metrics for comparring multiple models

I am building logistic regression models of some data to test a hypothesis and using cross-validation for each one. The cross-valdiation takes place within different subjects (mice in this case) of ...
Angus Campbell's user avatar
0 votes
1 answer
39 views

Model comparison: with raw or normalized data?

I have developed a index of drug addiction risk whose formula is Index = 1/log10(a_given_variable). The raw values of the calculated ...
M.S.'s user avatar
  • 1
3 votes
3 answers
270 views

Comparing models with main effects and interactions

I have two models: Model 1: Only contains independent variable $x$, while $x$ is non-significant. Model 2: Contains $x$, $m$, and $x * m$, and $x * m$ is significant. How could I illustrate this ...
david's user avatar
  • 31
0 votes
0 answers
24 views

Which test is appropriate when testing between nested groups

Imagine that we want to analyze the outcome of a variable $Y$. The variable $Y$ represents the number of test passed, going from 1 to 7. I want to know if $Y$ varies among different groups, for ...
lulufofo's user avatar
  • 482
9 votes
1 answer
83 views

Testing a multifactor asset pricing model against another one

I have a sample of $(Y_{i,t},X_{1,i,t},X_{2,i,t})$ for $i=1,\dots,N$ and $t=1,\dots,T$. I want to figure out which data generating process (DGP) it comes from, DGP1 or DGP2. DGP1: $$ Y_{i,t}=\lambda_0+...
Richard Hardy's user avatar
0 votes
0 answers
20 views

Interactions in regression and choosing appropriate model?

I’m running the following regression model, where MR is a continuous predictor and grade, gender, and race are all categorical. DV~MR + grade + gender + race I’m also interested in how the 3 ...
kmy221's user avatar
  • 11
0 votes
0 answers
144 views

Comparing the output distribution of two ML models

Consider a regression task (e.g. predicting house prices) with a given train and test sets. We start with constructing a linear regression model, in which we assume $y_i=X^T\beta+\epsilon$ with $E[\...
Spätzle's user avatar
  • 3,972
1 vote
6 answers
151 views

Regression model with good p-values but bad error

Suppose there is some data and all assumptions for regression are met. I fit 2 regression models now: Case 1: Regression model where all coefficients have significant $p$ values but overall the model ...
stats_noob's user avatar
2 votes
0 answers
68 views

Why don't deep learning-based computer vision studies utilize statistical tests for comparisons? [closed]

Many readers familiar with scientific articles in the field of deep learning-based computer vision might have observed a common practice: the absence of statistical significance tests for comparing ...
ricber's user avatar
  • 53
0 votes
1 answer
44 views

BIC model comparison [closed]

My go/nogo task was recorded in 4 blocks, each with a new set of cue images, with the learning and therefore the Q-values being reset at the beginning of each block. I have fitted different versions ...
Zahra's user avatar
  • 3
1 vote
1 answer
403 views

why are models compared over multiple datasets?

I cannot clearly understand why models are compared across multiple datasets. What practical problems do they aim to solve? Especially, in several papers, models are compared using datasets that are ...
a_burkley's user avatar
3 votes
1 answer
119 views

Compare non-linear model parameter estimates between two groups with random effects

I have a treatment (2 groups) for which I want to compare parameter estimate of a sigmoid curve (i.e., do any of the three parameters differ significantly between my two treatment group). I am ...
miki's user avatar
  • 294
0 votes
0 answers
27 views

How is it possible to compare powers of normality tests?

The famous paper [1] compares the power of several normality tests. Regardless of its result, my immediate question is, "How is that even possible?" Well, the definition of power in this ...
Student's user avatar
  • 235
0 votes
0 answers
84 views

Comparison of regression coefficients, same y but two different X versions

I've seen versions of this question but not specifically answering the question of a statistical test to compare the beta coefficients for two independent variables (x1, x2) but the same dependent ...
cmilando's user avatar
0 votes
0 answers
24 views

De Longs test on ROC curves?

I have two different ROC curves for Model A and B. I wanted to calculate the DeLongs test to identify the statisical significance of the difference between the model. model_A_tpr = [0, 0.2, 0.4, 0.6, ...
a_parida's user avatar
  • 101
6 votes
2 answers
193 views

How to compare fitted survival model with covariates vs. Kaplan-Meier?

How would one ideally compare fitted survival models with covariates vs. a Kaplan-Meier with a goal of getting an idea of whether the survival model describes the data well. E.g. in an example like ...
Björn's user avatar
  • 32.8k
1 vote
0 answers
19 views

Predictive Diagnostic, Comparison of simulated data with observed data

The question is quite abstract, so I display it with only the essential information. Suppose that we have three models $B_{1}, B_{2}$ and $F_{3}$. The $B_{1}, B_{2}$ are Bayesian models and the $F_{3}$...
Fiodor1234's user avatar
  • 2,254
4 votes
0 answers
169 views

Trying to understand the theory behind my similar / better results than XGBoost using a calibrated linear model (GAM)

I just opened a discussion on reddit asking about why/how the calibrated linear models I've been training have been getting similar / better results than XGBoost in my experiments. I was told to cross ...
William's user avatar
  • 41
1 vote
1 answer
49 views

Compare effects in a small sample

I would like to analyse whether measures A, B, and C have a different influence on a measure D compared to a measure X on D. My exploratory hypothesis is that A, B, or C could influence Z negatively ...
a.henrietty's user avatar
0 votes
1 answer
40 views

Assessing model performance [closed]

Suppose there are $n=100000$ locations. At each location,there are observations of dependent variable $Y$ for 30 years. That is, we have time series data and this data have $100000$ rows and $30$ ...
user149054's user avatar
1 vote
0 answers
19 views

Is it possible average an information criterion across models?

Is it possible to take the average of information criterion like the AIC? For my model comparison, I have 24 different models. I use 4 different GARCH models each with 6 different distributions for ...
Hello there's user avatar
1 vote
0 answers
14 views

Evaluate Value at Risk (VaR) models with different VaR backtesting approaches - averaging p-values?

I estimated the Value at Risk of a time series of log returns with different approaches and models. Now I want to compare the models and choose the model that most accurately estimated the Value at ...
Isabel's user avatar
  • 31
0 votes
0 answers
17 views

Choosing one method for modeling Value at Risk (VaR) over another - determining the "best" model

I estimated the Value at Risk of a time series of log returns with different approaches and models. Now I want to compare the models and chose the model that most accurately estimated the Value at ...
Isabel's user avatar
  • 31
0 votes
0 answers
28 views

Multiple regression and post-hoc analysis

Dependent Variable(s): cognitive test Independent Variable: Sex (covariate), Age (covariate), working memory, long-term memory test, interaction(working memory: long-term memory) However, the ...
Tube's user avatar
  • 115
1 vote
1 answer
35 views

Comparing different methods for modeling Value at Risk (VaR) - understanding VaR backtesting results

I estimated the Value at Risk of a time series of log returns with different approaches and models. Now I want to compare the models and find the model that most accurately estimated the Value at Risk....
Isabel's user avatar
  • 31
0 votes
0 answers
44 views

comparison of generalize linear mixed effect models in R with compare_performance function

Below I add my model comparison between a random intercept and random slope model using performance library. I used compare_performance. My models are: ...
Balina's user avatar
  • 5
1 vote
1 answer
108 views

How to Choose Polynomial Degree for Regression Model when Error Keeps Reducing As Degree Increase

I have a relatively small dataset with less than 100 samples, with one predictor and one outcome variable, both numerical. I generated models using lm and glm functions. For linear and polynomial (2 ...
Patrick C.'s user avatar
1 vote
0 answers
37 views

Is the Wilcoxon Signed Rank Test appropriate when the Brier score is the accuracy metric?

When comparing model performance, is it valid to use the Wilcoxon signed rank test for matched pairs, when the accuracy metric is the Brier score? (Here, the Brier score is used in calculating the OOB ...
Kyle's user avatar
  • 363
1 vote
1 answer
68 views

Comparing binary and continuous predictors for diagnostic test

There is a "traditional" biomarker (binary predictor) used in the diagnosis of a disease (binary outcome) that has a high cost to perform for the clinical labs. I'm studying alternatives, ...
Daniel Nunes's user avatar
0 votes
0 answers
22 views

Can I compare forecasting performance of rolling window VAR and usual forecast of model with ARIMA errors?

Can I compare forecasting performance of rolling window VAR and usual forecast of model with ARIMA errors? Or maybe there is exist better way to compare forecasting performance?
Arri's user avatar
  • 47
0 votes
0 answers
87 views

How to compare the performance of a volatility forecast like GARCH (1,1) with exogenous variables (MSE?)

I want to investigate, weather financial news have an influence on the volatility prediction of asset returns (daily data) when including them into the variance model/mean model. I have fit a GARCH/...
Jascäcilie's user avatar
5 votes
2 answers
386 views

When dealing with data imbalance, shouldn't we never compare models based on validation loss, or at least weight it?

I know that when validating we are interested in knowing how the model performs in real-world scenarios, so we want the class ratios during validation/test to be the original ones. Say, however, that ...
raquelhortab's user avatar
0 votes
1 answer
78 views

Statistically testing MSE difference between models predictions without normality of squared residuals

I have a dataset that I have splitted for training - testing purpose (~2400 - ~600) . After training an XGBoost regression model on the training set, I would like to statistically test the fact that ...
Guilhem Nespoulous's user avatar
1 vote
0 answers
76 views

How to compare AI model with other models for research purposes

If I have X and Y datasets, I will apply meta-learning few-shot-learning by training the model on the X dataset and applying meta-testing using the Y dataset. While I compare my result with other ...
AMAS AL's user avatar
  • 33
3 votes
1 answer
129 views

Why does the Type-3 ANOVA using LRT via car::Anova() give different result than term-by-term LRT model comparison via anova() in R?

With this very simple data: ...
Couchiagna's user avatar

1
2 3 4 5
12