# Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

377 questions
Filter by
Sorted by
Tagged with
28 views

### What inference can we get from the frequency distribution of classification thresholds?

I've the probability scores of positive class of two models. The frequency distribution of those probability scores(thresholds) are like this Model #1 Model #2 It's a binary classification problem....
16 views

### Why do the car::Anova(test=“LRT”, type=3) and anova(test=“LRT”) slightly differ in the output (details inside) [duplicate]

Let's say we have this data: ...
33 views

### Comparing mixed models with weighted variance

I'm performing some linear mixed models for a psychological experiment. I'm not a statistician so my knowledge is limited. The basic idea is that: I have an experiment in which I model my response ...
64 views

### Comparing different model with the same output

I work on developing 3 different models for predicting land surface temperatures in an area (validation is done with images taken by a thermal drone). The models are: A thermodynamic mechanistic ...
29 views

### How do I compare different regression models?

I have some data and after doing both LASSO and least squares estimation (the usual multiple linear regression) on the data I want to compare the performance of these models. If I compare the squared ...
53 views

### Is this the correct way of working on an ML problem?

I am working on an assignment where I have been provided a data-set using which I have to do the following: A. The data in the F20 column is missing, I have to use various imputation strategies and ...
52 views

### How to statistically compare machine learning “regression” models?

Let say that I want to compare the performance of XGBoost vs NN, or NN vs NN, or even the same NN at different epochs for a regression task. All algorithms are trained and evaluated on the exact ...
26 views

### compare correlation between two matrices

We have two algorithms that assign a score to the foreground of how well it fits the background. And we want to know which algorithm is closer to human perception. Then we collect an evaluation set ...
13 views

### How to compare multiple regression models

I'm building different supervised learning models in order to predict house pricing: linear regression, random forest, xgbost, etc My question is: how do I compare the results for this models? What ...
19 views

### Comparison of ARIMA and VAR accuracy

Can someone help explaining how to compare a forecast from an ARIMA model and a VAR model. I have tried calculating MAPE, MSE, RMSE etc. for my VAR forecast, but i simply cannot get it to work. ...
15 views

### Passive Combination of two Classifiers at the level of Class Labels

I have three classifiers - 1. Vision based classifier trained to detect class labels such as pedestrian and cars 2. Radar based classifier to detect same class labels 3. Lidar based classifiers to ...
36 views

### Model selection based on the closest fit to another model

For each subject I have two measures, let's say $X$ and $Y$, that correspond to the same measurement but each from a different measurement device. In theory we assume that ideally these two ...
17 views

### Why is pseudo-Bayesian Model Averaging using WAIC giving counterintuitive results for my model? Issue with unconstrained data uncertainty?

I'm currently working on a model which fits a pair of curves which are parameterized by some shape and scaling parameters. I want to produce weights for two different models. In my first model, which ...
10 views

### Bayesian Inference: Predicting Model Validity From Model Evidence

Bayes Theorem states that $$P[\theta | D] = \frac{P[D|\theta]P[\theta]}{P[D]}$$ where model evidence is defined as $$P[D] = \int P[D|\theta]P[\theta]d\theta$$ I have learned that model evidence ...
8 views

### What causes a chol.default(tmpvc) error in the vuong test when comparing non nested models?

I have a series of ordinal logistic regressions. Each predicts the same outcome (Y), and each has a single predictor (X1, X2 or X3 (All are strongly correlated)). I want to determine whether any of ...
15 views

### Compare two classifiers that works on two different datasets

I have 2 activity-recognition classifiers working on two different dataset (representing different repetitions of the same movement). Since the performance values are obtained from different samples, ...
23 views

### Is there an AIC and BIC equivalent for MAP?

I would like to know if there is an AIC or BIC equivalent for maximum a posterior estimation. I'm trying to compare several different models that have been fit using MAP, but I am unsure of the best ...
70 views

### Alternatives for Diebold-Mariano test when comparing the best forecast among many against a benchmark

Suppose I encounter a new forecasting method and I wish to evaluate it against a benchmark. I can obtain forecasts from the two methods and compare them to actual realizations and thus obtain the ...
101 views

### Dietterich's 5x2cv paired t-test for regression problems

In 1998 Dietterich proposed an algorithm for comparing classification algorithms using a series of cross validated t-tests: Dietterich, Thomas G. "Approximate statistical tests for comparing ...
71 views

### K-Fold Cross Validation, the right way

I searched for this in many forums, but didn't find any answers that would answer my question (or I didn't understand correctly). So, I will post here. I have a dataset A I have a machine learning ...
30 views

### Is the AIC of an ARIMA model comparable with the AIC of a regression with ARIMA errors?

I already know that models from different "families" cannot be compared through AIC (or other information criteria such as BIC or HQIC), however, I'm not sure about the specific case of ARIMA and ...
30 views

### compare coefficients from different regression

I estimate the following models using the Hausman-Taylor estimator: $$y_{i,t} = a_{0} + B_1 controls_{i,t} + \beta_1x_{i,t=2000} + B_2 Year_t + B_3 x_{i,t=2000}*Year_t + e_{i,t}, (1)$$ y_{i,...
8 views

### Questions Tags Users Unanswered Connection of the critical value for KS distance and model complexity

I asked a similar question is math exchange, but here it seems to fit better. I wonder about a relationship of the critical value for the KS distance (with Lilliefor correction) and model complexity? ...
58 views

### Nested Cross-Validation with Test Set (Model Selection)

I have seen many examples on Nested CV where someone takes the entire dataset and performs nested CV on it. My question is: for model comparison shouldn't we first split the original dataset into ...
33 views

### What are some of the most correct/accepted ways to tune and compare different models in an academic context?

Those days, I have been reviewing different academic papers which mainly compare the performance of different machine learning methods on a particular problem. And I was surprised by the variety of ...
15 views

### Bayesian Comparison of Dependent Correlation Coefficients

I would like to compare the correlation coefficients from two different models of the same data points. One of them is the raw bivariate data (Value vs Time), and the other is given a biologically-...
58 views

### Is it valid to use Anova (in R) to compare alternative multinomial log-linear models?

I am familiar with the idea of comparing alternative linear regression models using anova(model1,model2), for models fitted using ...
24 views

### Choosing the best aggregation location measure

I'm trying to analyze a granular dataset that is composed as follows: ...
96 views

### WAIC for model comparisons--overly conservative?

I'm having a hard time wrapping my head around the relationship between model posterior predictions and model comparisons via WAIC. Specifically, how do I interpret findings where a model including ...
264 views

### Should I cross-validate metrics that were not optimised?

I want to compare two models. Say I have two objective functions on the same data $f(X,y,\theta)$ and $g(X,y,\theta)$ that both evaluate the models performance in ways that I am interested in ($\theta$...
12 views

### How to compare our model with state-of-the-art deep models which are given for other frameworks?

I am preparing a paper presenting a new deep model. I wonder how to compare our model with the state-of-the-art models. In other papers, authors used the same dataset but not with same data split (...
125 views

### Can I use the AIC to compare ordered logistic regression models based on different datasets?

I performed ordered logistic regression using the polr() command in R. I did this with the same independent and dependent variable (the X and Y are the same for every model), but the exact data in ...
10 views

### Reporting difference in distributions for text

I am working on a problem that can be translated to determining which of two corpora generated a given sample text. I have a hypothesis for before some processing and other for after processing, and ...
18 views

### DIC to compare models with different numbers of parameter?

I am interested in comparing hierarchical Bayesian models based on the same dataset but differing in their spatial and temporal resolution. In short, I am looking at animal population changes over ...
40 views

### Comparing two machine learning models

Let's say I'm working with two models, ResNet-50 and ResNext-50, and I want to compare the results. Due to the stochastic nature of deep learning, would it be advantageous or even encouraged to seed ...
22 views

### Loss function for one-step-ahead volatility forcasts

I'm trying to perform the MCS test using the R-package "MCS" to compare GARCH-MIDAS Models. The loss function requires as inputs a vector with some realized volatility measure ˜ σt+1 (I chose the ...
19 views

### Difference between null and saturated log-linear models

I have data from an experiment testing the number of 'cases' at each of three measurement points (0, 12, and 24 weeks). I am interested in whether the proportion of cases across these measurement ...
26 views

### Likelihood ratio to quantify the similarity between one sample with two other matched samples

I conducted a study with 3 conditions and N subjects. All subjects performed all conditions once. I would like to know if the first condition is similar to the second or third condition. Formally, ...
142 views

### Bayesian methods are about averaging over uncertainty rather than optimization. Explain?

I came across the statement "The key ingredient in Bayesian methods is to average over your uncertain variables and parameters, rather than to optimize". Can someone explain why this is? ...
140 views

### Bayes factors and predictive accuracy in model comparison in rstan / brms

Despite reading up on the subject, I can't wrap my head round it, so the question remains on shaky grounds, and responses along the lines of "read chapter x" are very welcome. What I'm doing is I'm ...
260 views

### NLS model comparisons by ANOVA

I am comparing NLS models from two groups, males and females. I compared the models by ANOVAs, but I am hesitating which is the correct model comparison to use: a) Subgroup Males vs Subgroup Females ...
10 views

### Deterioration of the accuracy of a system over time

I have a system that compares the predicted variable with the true variable by calculating the absolute error percentages. . Where $\pi$ is the predicted variable and P is the true variable. And you ...
35 views

### How to compare two classifications, visually?

I have two classifiers, and I was asked (for learning purposes) to compare their predictions, visually. The target value is a real value (sentiment) What could be a good way to compare them?
12 views

### Marginal effect from two models with different dependent variables

This question is somehow related to this one and this one. I have two models with the same independent variables on the same dataset. Model1: Y1 = alpha0 + alpha1 X1 + alpha2 X2 + error1 Model2: ...
120 views

### Permutation test comparing nested non-linear models with an exchangeable dummy variable

This question is closely related to an earlier question, but I realized my case was actually a lot more specific than the way I formulated it there, in ways that I think merit a separate answer. I ...
10 views

### Model comparison with new IV over data subset: to rebuild base model or not

I'm modeling a few DVs with a set of IVs via glmer. I have successfully built a drop-1 nested model comparison framework to test the significance of each variable. ...
346 views

### Interpretation of R output from Cohen's Kappa

I have the following result from carrying out Cohen's kappa in R ...
250 views

### How to interpret/compare R2 score(s)? [duplicate]

I know that an R² score of 1 is a perfect fit of the model to the truth, a 0 is an constant output regardless of the input, and that negative values are possible when the output varies, but there is ...