# Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

245 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
718 views

### How to calculate percent partial deviance explained by each predictor variable in a GAM model?

I am trying to find a sensible way to calculate the deviance explained by each predictor variable in a GAM model and need some input on my calculations. Following Simon Wood's example on the thread ...
• 18.7k
989 views

### Are Log Predictive Likelihood, Log Predictive Probability, Log Marginal Likelihood and Log Predictive Density same?

I have seen different papers use different terms to express the scoring rules that they used to compare Bayesian models. Some of those terms are, Log Predictive Density (Bayesian Data Analysis - by ...
• 153
619 views

### F-test to determine whether more than two sets of data differ

Here is the context for my question: I understand that you can fit the same model to two different datasets separately and then fit the model to the datasets pooled together as a way to discern ...
• 510
1k views

### How to compare models with different distributional assumptions for response variable in GLM?

Let's say I have measurements $Y$ which are all positive, and the distribution seems to be somewhat skewed. I'm modelling $Y$ in GLM framework. Now I could set my GLM using different distributional ...
• 136
786 views

### When to use likelihood ratio test vs. incremental F-test in nested nonlinear model selection?

I have two nested nonlinear models and I want to know which provides a better fit to some data. I see descriptions of both the likelihood ratio test and of the incremental F-test (also called the ...
4k views

### AIC or ANOVA to compare models?

What are the relative merits of each approach, and which circumstances call for one rather than the other? To some extent I have a specific example in mind, which I've discussed here. In that example ...
256 views

### AIC Comparison for MLM with Different Distributions

Thank you in advance for your time and consideration! I am a non-mathematically-inclined graduate student in communication just learning multilevel modeling. We are running different models - some ...
348 views

### Compare and quantify relative improvement in ROC AUC scores?

What is an appropriate method for comparing relative improvement in model performance across different problems? For example, say I have three different datasets/problems a, b, c, and two models for ...
508 views

### Different goodness-of-fit statistics for same model fit as GLM and linear mixed model - why?

When I fit a simple linear model such as y ~ 1 + x1 + x2 in MATLAB R2018a or R2017a, the fitglm() and ...
493 views

### Connection between log predictive density and Kullback-Leibler information measure

I have read in Bayesian Data Analysis by Andrew Gelman that log predictive density can be used to compare Bayesian models due to its connection to the Kullback-Leibler information. The log ...
• 153
4k views

### The anova() function in R to compare a glm() and a glmer() models

I recently tested something regarding the stats::anova function in R. Assuming modglm is a model fitted with the ...
• 141
708 views

### Am I calculating the Anderson-Darling statistic correctly for comparing PDFs?

I want to compute the Anderson-Darling statistic to use as a goodness of fit test for some PDF distributions. I am using R. I want to compare some models to the observations, which are my baseline. ...
• 91
600 views

### Approximating the marginal likelihood in Bayesian Model Comparison

Given some data $y$, my interest centers around a collection of models $\{\mathcal{M}_1,\mathcal{M}_2,\cdots,\mathcal{M}_L\}$ representing competing hypotheses about $y$. Each model $\mathcal{M}_l$ ...
• 3,846
2k views

### model comparison: chi2 vs. AIC (and BIC)

I've been using the approach from Appendix A from this paper. It states that if I have two fits with $\chi^2$ values $\chi^2_1(\nu_1)$ and $\chi^2_2(\nu_2)$ ($\nu$ - degree of freedom), then their ...
• 1,155
184 views

### Evaluating survival models in the presence of covariate-dependent censoring

I have a censored survival analysis problem with the following characteristics: Failure times are discretized The censorship distribution depends on certain covariates I don't have a properly-...
• 5,508
3k views

### compare quasi poisson models

I have two models: ...
• 115
2k views

### Likelihood ratio test versus AIC for model comparison

When performing model comparison, why does the likelihood ratio test require two nested models while this is not required when using the AIC?
• 1,737
446 views

### Comparing two non-linear biological models: differences in y vs. differences in slope

I'm an ecology PhD student, but unfortunately not a very mathematically sophisticated one. I am interested in ways to compare two non-linear models. I was really impressed with this answer, which ...
• 93
379 views

### Which statistical test to compare same model with different parameters?

I have two datasets on people buying apples based on weight and price. One dataset in 2019 the other in 2020. I estimate a logit model with Utility = betaWeight * weight + betaPrice * price. Training ...
• 31
91 views

### AIC Debate Once Again...Comparison of Likelihoods Across Distributions

I am following up on this post Is it okay to compare fitted distributions with the AIC?. I've looked at the document AIC Myths. I still am not satisfied. Context: I'm working on a project in ...
• 151
326 views

### WAIC for model comparisons--overly conservative?

I'm having a hard time wrapping my head around the relationship between model posterior predictions and model comparisons via WAIC. Specifically, how do I interpret findings where a model including ...
• 31
338 views

### How do I create a lift curve?

I am having trouble constructing a lift curve from scratch. Say I have bunch of 0-1 labels, and predictive labels. In order to construct the lift curve, I do the following steps (assume I have ...
• 215
468 views

### Effect size for comparison of two nested models

I want to calculate an effect size for the difference of two structural equation models (model 1 and 2). Model 2 is nested in model 1. For each model I get a $\chi^2$ as measure of fit. Whether the ...
• 627
410 views

### Compare GLM with different dependent variables (that have not the same variance)

I would like to know if it is possible to compare the effects of one predictor on different response variables that follow e.g. a Poisson distribution. I have already read this topic with excellent ...
• 253
312 views

### AIC comparison for models without maximum likelihood

AIC is a popular model comparison measure (despite of its potential shortcomings). I am wondering whether it is legit to compare AIC (or Akaike weights) for models that were fitted without requiring ...
• 486
173 views

### Comparing linear models

My Linear Models professor said that it is possible sometimes to obtain two different linear models for the same data, such that the models cannot be compared to determine which is better. How can ...
• 1,221
287 views

### Comparing model fit with heteroscedastic data

I am developing a physiological test using R that requires some parameters optimised. In comparing the new method against the existing method, the values of individual readings correlate in a linear ...
25 views

### Model selection: comparing Bayesian models with likelihood vs likelihood-free (Approximate Bayesian Computation)

I have two families of models that can possibly explain the data at hand. One family is rather process-based, using fairly complicated simulations and Approximate Bayesian Computation to estimate the ...
• 121
490 views

### Differences between time-series regression versus cross-sectional regression

I am currently trying to grasp the differences between time series regression and cross-sectional regression, because these terms are often used in papers about market predictability. I understand ...
• 31
84 views

### Can I use a (paired) t-test to compare the accuracies of classification models on a hold out test set?

When comparing the Brier scores or logloss/cross-entropy for multiple models on a hold-out test set, how can I be sure the 'best' model doesn't just have the lowest loss due to chance? When can I be ...
• 2,612
27 views

### Is there a way to analyse main effects on quantile mixed-effect regression in the spirit of the "ANOVA" procedure (now for medians)?

I would like to analyse my data using quantile regression with random effect. The problem is I have also categorical covariates, which will "split" the output into corresponding levels. I ...
• 103
56 views

### Develop granularity-invariant criteria for comparison of logistic (binomial) models

I have a model with logistic (binomial) likelihood, with number of successes and failures as a response variable. I am comparing various models, which can be of different granularity. Different ...
• 5,925
66 views

### Comparison of volatility models - using conditional standard deviations or conditional variances?

At the below content, I learned that; "an unbiased variance estimator's square root doesn't imply being an unbiased estimator of the standard deviation". Comparison of daily fitted volatility and ...
• 639
29 views

### Different cut aria effect in aligning and comparing two STL file with cloudcompare

I have one project for dental thesis which some aspect of it is not clear for me. In this project we need to compare one specific area of dental mold scan by cloudcompare software, so the area is ...
132 views

### Why is pseudo-Bayesian Model Averaging using WAIC giving counterintuitive results for my model? Issue with unconstrained data uncertainty?

I'm currently working on a model which fits a pair of curves which are parameterized by some shape and scaling parameters. I want to produce weights for two different models. In my first model, which ...
• 21
56 views

### What are some of the most correct/accepted ways to tune and compare different models in an academic context?

Those days, I have been reviewing different academic papers which mainly compare the performance of different machine learning methods on a particular problem. And I was surprised by the variety of ...
• 45
211 views

• 954
56 views

### Is augmentation of treatment effect caused significantly by mediator?

I want to find out if the influence of potential mediator P on treatment effect D is significant. In the two models below $\beta_1$ and $\lambda_1$ are different and I don't know how to test if the ...
• 566
124 views

### Is it possible to have a lower AIC value for the smaller of two nested models?

I am testing the parallel lines assumption of an ordered probit model by running a likelihood-ratio test against a model where the coefficients can vary for each step of the ordinal response variable. ...
370 views

### References for "self-fulfilling prophecy bias" in machine learning needed

I am currently looking for a scientific reference (journal article, book etc) that describes the challenge of evaluating the performance of a method on the data it produces itself. For example: A ...
• 63
37 views

### Method for selecting interactions of interest

I have a set of 9 predictors from which I disregarded 3 in search of the best possible model (that with the highest $R^2$), so I have left with 6 independent variables. Now I want to find the best ...
• 109
165 views

### Power analysis for model comparison between linear mixed-effects models

I am trying to model the effect of 3 continuous predictors on a binary choice outcome. These data come from an experiment in which multiple subjects were ran in a task containing multiple trials (...
• 55
744 views

### Binary classification Confidence Interval AUC using positive/negative samples

For my thesis I would like to compare different supervised machine learning algorithms based on their AUC score. My supervisor pointed out to also report the confidence intervals (CI) of the AUC ...
• 21
3k views

### logic of resamples function in caret R package

The caret R package includes functions to compare models via their resampling distributions. Specifically, it prescribes fitting multiple models using the same resampling profiles (i.e. same versions ...
514 views

### AIC (QIC) - comparision of models?

I am comparing models created by geeglm function in R software. Does QIC of a model need to be lower than QIC of a null model? (in geeglm, geepack, R)
• 63