Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

245 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
10 votes
0 answers
718 views

How to calculate percent partial deviance explained by each predictor variable in a GAM model?

I am trying to find a sensible way to calculate the deviance explained by each predictor variable in a GAM model and need some input on my calculations. Following Simon Wood's example on the thread ...
8 votes
0 answers
989 views

Are Log Predictive Likelihood, Log Predictive Probability, Log Marginal Likelihood and Log Predictive Density same?

I have seen different papers use different terms to express the scoring rules that they used to compare Bayesian models. Some of those terms are, Log Predictive Density (Bayesian Data Analysis - by ...
  • 153
7 votes
0 answers
619 views

F-test to determine whether more than two sets of data differ

Here is the context for my question: I understand that you can fit the same model to two different datasets separately and then fit the model to the datasets pooled together as a way to discern ...
  • 510
7 votes
0 answers
1k views

How to compare models with different distributional assumptions for response variable in GLM?

Let's say I have measurements $Y$ which are all positive, and the distribution seems to be somewhat skewed. I'm modelling $Y$ in GLM framework. Now I could set my GLM using different distributional ...
  • 136
6 votes
0 answers
786 views

When to use likelihood ratio test vs. incremental F-test in nested nonlinear model selection?

I have two nested nonlinear models and I want to know which provides a better fit to some data. I see descriptions of both the likelihood ratio test and of the incremental F-test (also called the ...
5 votes
0 answers
4k views

AIC or ANOVA to compare models?

What are the relative merits of each approach, and which circumstances call for one rather than the other? To some extent I have a specific example in mind, which I've discussed here. In that example ...
4 votes
0 answers
256 views

AIC Comparison for MLM with Different Distributions

Thank you in advance for your time and consideration! I am a non-mathematically-inclined graduate student in communication just learning multilevel modeling. We are running different models - some ...
4 votes
0 answers
348 views

Compare and quantify relative improvement in ROC AUC scores?

What is an appropriate method for comparing relative improvement in model performance across different problems? For example, say I have three different datasets/problems a, b, c, and two models for ...
4 votes
0 answers
508 views

Different goodness-of-fit statistics for same model fit as GLM and linear mixed model - why?

When I fit a simple linear model such as y ~ 1 + x1 + x2 in MATLAB R2018a or R2017a, the fitglm() and ...
4 votes
0 answers
493 views

Connection between log predictive density and Kullback-Leibler information measure

I have read in Bayesian Data Analysis by Andrew Gelman that log predictive density can be used to compare Bayesian models due to its connection to the Kullback-Leibler information. The log ...
  • 153
4 votes
0 answers
4k views

The anova() function in R to compare a glm() and a glmer() models

I recently tested something regarding the stats::anova function in R. Assuming modglm is a model fitted with the ...
  • 141
4 votes
0 answers
708 views

Am I calculating the Anderson-Darling statistic correctly for comparing PDFs?

I want to compute the Anderson-Darling statistic to use as a goodness of fit test for some PDF distributions. I am using R. I want to compare some models to the observations, which are my baseline. ...
  • 91
4 votes
0 answers
600 views

Approximating the marginal likelihood in Bayesian Model Comparison

Given some data $y$, my interest centers around a collection of models $\{\mathcal{M}_1,\mathcal{M}_2,\cdots,\mathcal{M}_L\}$ representing competing hypotheses about $y$. Each model $\mathcal{M}_l$ ...
4 votes
0 answers
2k views

model comparison: chi2 vs. AIC (and BIC)

I've been using the approach from Appendix A from this paper. It states that if I have two fits with $\chi^2$ values $\chi^2_1(\nu_1)$ and $\chi^2_2(\nu_2)$ ($\nu$ - degree of freedom), then their ...
  • 1,155
4 votes
0 answers
184 views

Evaluating survival models in the presence of covariate-dependent censoring

I have a censored survival analysis problem with the following characteristics: Failure times are discretized The censorship distribution depends on certain covariates I don't have a properly-...
  • 5,508
4 votes
0 answers
3k views

compare quasi poisson models

I have two models: ...
4 votes
0 answers
2k views

Likelihood ratio test versus AIC for model comparison

When performing model comparison, why does the likelihood ratio test require two nested models while this is not required when using the AIC?
  • 1,737
4 votes
0 answers
446 views

Comparing two non-linear biological models: differences in y vs. differences in slope

I'm an ecology PhD student, but unfortunately not a very mathematically sophisticated one. I am interested in ways to compare two non-linear models. I was really impressed with this answer, which ...
3 votes
2 answers
379 views

Which statistical test to compare same model with different parameters?

I have two datasets on people buying apples based on weight and price. One dataset in 2019 the other in 2020. I estimate a logit model with Utility = betaWeight * weight + betaPrice * price. Training ...
  • 31
3 votes
0 answers
91 views

AIC Debate Once Again...Comparison of Likelihoods Across Distributions

I am following up on this post Is it okay to compare fitted distributions with the AIC?. I've looked at the document AIC Myths. I still am not satisfied. Context: I'm working on a project in ...
  • 151
3 votes
0 answers
326 views

WAIC for model comparisons--overly conservative?

I'm having a hard time wrapping my head around the relationship between model posterior predictions and model comparisons via WAIC. Specifically, how do I interpret findings where a model including ...
  • 31
3 votes
0 answers
338 views

How do I create a lift curve?

I am having trouble constructing a lift curve from scratch. Say I have bunch of 0-1 labels, and predictive labels. In order to construct the lift curve, I do the following steps (assume I have ...
  • 215
3 votes
0 answers
468 views

Effect size for comparison of two nested models

I want to calculate an effect size for the difference of two structural equation models (model 1 and 2). Model 2 is nested in model 1. For each model I get a $\chi^2$ as measure of fit. Whether the ...
  • 627
3 votes
0 answers
410 views

Compare GLM with different dependent variables (that have not the same variance)

I would like to know if it is possible to compare the effects of one predictor on different response variables that follow e.g. a Poisson distribution. I have already read this topic with excellent ...
  • 253
3 votes
0 answers
312 views

AIC comparison for models without maximum likelihood

AIC is a popular model comparison measure (despite of its potential shortcomings). I am wondering whether it is legit to compare AIC (or Akaike weights) for models that were fitted without requiring ...
  • 486
3 votes
0 answers
173 views

Comparing linear models

My Linear Models professor said that it is possible sometimes to obtain two different linear models for the same data, such that the models cannot be compared to determine which is better. How can ...
  • 1,221
3 votes
0 answers
287 views

Comparing model fit with heteroscedastic data

I am developing a physiological test using R that requires some parameters optimised. In comparing the new method against the existing method, the values of individual readings correlate in a linear ...
2 votes
0 answers
25 views

Model selection: comparing Bayesian models with likelihood vs likelihood-free (Approximate Bayesian Computation)

I have two families of models that can possibly explain the data at hand. One family is rather process-based, using fairly complicated simulations and Approximate Bayesian Computation to estimate the ...
  • 121
2 votes
0 answers
490 views

Differences between time-series regression versus cross-sectional regression

I am currently trying to grasp the differences between time series regression and cross-sectional regression, because these terms are often used in papers about market predictability. I understand ...
  • 31
2 votes
0 answers
84 views

Can I use a (paired) t-test to compare the accuracies of classification models on a hold out test set?

When comparing the Brier scores or logloss/cross-entropy for multiple models on a hold-out test set, how can I be sure the 'best' model doesn't just have the lowest loss due to chance? When can I be ...
  • 2,612
2 votes
0 answers
27 views

Is there a way to analyse main effects on quantile mixed-effect regression in the spirit of the "ANOVA" procedure (now for medians)?

I would like to analyse my data using quantile regression with random effect. The problem is I have also categorical covariates, which will "split" the output into corresponding levels. I ...
2 votes
1 answer
56 views

Develop granularity-invariant criteria for comparison of logistic (binomial) models

I have a model with logistic (binomial) likelihood, with number of successes and failures as a response variable. I am comparing various models, which can be of different granularity. Different ...
  • 5,925
2 votes
0 answers
66 views

Comparison of volatility models - using conditional standard deviations or conditional variances?

At the below content, I learned that; "an unbiased variance estimator's square root doesn't imply being an unbiased estimator of the standard deviation". Comparison of daily fitted volatility and ...
  • 639
2 votes
0 answers
29 views

Different cut aria effect in aligning and comparing two STL file with cloudcompare

I have one project for dental thesis which some aspect of it is not clear for me. In this project we need to compare one specific area of dental mold scan by cloudcompare software, so the area is ...
2 votes
0 answers
132 views

Why is pseudo-Bayesian Model Averaging using WAIC giving counterintuitive results for my model? Issue with unconstrained data uncertainty?

I'm currently working on a model which fits a pair of curves which are parameterized by some shape and scaling parameters. I want to produce weights for two different models. In my first model, which ...
  • 21
2 votes
1 answer
56 views

What are some of the most correct/accepted ways to tune and compare different models in an academic context?

Those days, I have been reviewing different academic papers which mainly compare the performance of different machine learning methods on a particular problem. And I was surprised by the variety of ...
  • 45
2 votes
0 answers
211 views

How do you / can you compare Bayesian vs. frequentist regression models?

I am working on a regression model to predict a target variable in a dataset with over 100 features. Three different regression models are defined and fit in order to compare their performance using $...
  • 21
2 votes
0 answers
623 views

How to compare non-nested models (with different number of items) in CFA?

I investigate the factor structur of an existing scale in a data set with N = 714 (using lavaan, R). In this context I would like to compare to alternative models: Model 1: Two factors with 5 items ...
2 votes
1 answer
825 views

How to compare model fit of OLS and Poisson regression?

I have built two regression models to predict sales of different products based on a number of explanatory variables, with an offset term for the number of days each product was on sale. One is a ...
2 votes
0 answers
578 views

Accuracy metric for comparing Time Series models?

I'm writing a blog post on forecasting time series with autoregression. In it, I compare the performance of SLR, ARIMA, and SARIMAX on forecasting the number of Home Sales in Seattle (see below). All ...
2 votes
0 answers
1k views

Comparing log-transformed and non-log transformed models

Suppose I have the following two models fitted with constrained linear least squares: Model 1: $Y = \beta_1X_1 + \ldots + \beta_kX_k + \varepsilon$ Model 2: $\log_{10}(Y) = \beta_1\log_{10}(X_1) + \...
  • 954
2 votes
1 answer
56 views

Is augmentation of treatment effect caused significantly by mediator?

I want to find out if the influence of potential mediator P on treatment effect D is significant. In the two models below $\beta_1$ and $\lambda_1$ are different and I don't know how to test if the ...
  • 566
2 votes
1 answer
124 views

Is it possible to have a lower AIC value for the smaller of two nested models?

I am testing the parallel lines assumption of an ordered probit model by running a likelihood-ratio test against a model where the coefficients can vary for each step of the ordinal response variable. ...
2 votes
0 answers
370 views

References for "self-fulfilling prophecy bias" in machine learning needed

I am currently looking for a scientific reference (journal article, book etc) that describes the challenge of evaluating the performance of a method on the data it produces itself. For example: A ...
2 votes
0 answers
37 views

Method for selecting interactions of interest

I have a set of 9 predictors from which I disregarded 3 in search of the best possible model (that with the highest $R^2$), so I have left with 6 independent variables. Now I want to find the best ...
  • 109
2 votes
0 answers
165 views

Power analysis for model comparison between linear mixed-effects models

I am trying to model the effect of 3 continuous predictors on a binary choice outcome. These data come from an experiment in which multiple subjects were ran in a task containing multiple trials (...
  • 55
2 votes
0 answers
744 views

Binary classification Confidence Interval AUC using positive/negative samples

For my thesis I would like to compare different supervised machine learning algorithms based on their AUC score. My supervisor pointed out to also report the confidence intervals (CI) of the AUC ...
  • 21
2 votes
0 answers
3k views

logic of resamples function in caret R package

The caret R package includes functions to compare models via their resampling distributions. Specifically, it prescribes fitting multiple models using the same resampling profiles (i.e. same versions ...
2 votes
0 answers
514 views

AIC (QIC) - comparision of models?

I am comparing models created by geeglm function in R software. Does QIC of a model need to be lower than QIC of a null model? (in geeglm, geepack, R)
  • 63
2 votes
0 answers
101 views

Model comparison using lower bound from variational approximation

I applied variational approximation for probit regression model and got the lower bound for the log marginal likelihood. When I compare models with different covariates using lower bound, I found that ...
  • 23

1
2 3 4 5