Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

Filter by
Sorted by
Tagged with
0
votes
0answers
28 views

What inference can we get from the frequency distribution of classification thresholds?

I've the probability scores of positive class of two models. The frequency distribution of those probability scores(thresholds) are like this Model #1 Model #2 It's a binary classification problem....
1
vote
1answer
33 views

Comparing mixed models with weighted variance

I'm performing some linear mixed models for a psychological experiment. I'm not a statistician so my knowledge is limited. The basic idea is that: I have an experiment in which I model my response ...
2
votes
1answer
64 views

Comparing different model with the same output

I work on developing 3 different models for predicting land surface temperatures in an area (validation is done with images taken by a thermal drone). The models are: A thermodynamic mechanistic ...
1
vote
0answers
29 views

How do I compare different regression models?

I have some data and after doing both LASSO and least squares estimation (the usual multiple linear regression) on the data I want to compare the performance of these models. If I compare the squared ...
0
votes
0answers
53 views

Is this the correct way of working on an ML problem?

I am working on an assignment where I have been provided a data-set using which I have to do the following: A. The data in the F20 column is missing, I have to use various imputation strategies and ...
3
votes
0answers
52 views

How to statistically compare machine learning “regression” models?

Let say that I want to compare the performance of XGBoost vs NN, or NN vs NN, or even the same NN at different epochs for a regression task. All algorithms are trained and evaluated on the exact ...
0
votes
1answer
26 views

compare correlation between two matrices

We have two algorithms that assign a score to the foreground of how well it fits the background. And we want to know which algorithm is closer to human perception. Then we collect an evaluation set ...
1
vote
1answer
13 views

How to compare multiple regression models

I'm building different supervised learning models in order to predict house pricing: linear regression, random forest, xgbost, etc My question is: how do I compare the results for this models? What ...
0
votes
0answers
19 views

Comparison of ARIMA and VAR accuracy

Can someone help explaining how to compare a forecast from an ARIMA model and a VAR model. I have tried calculating MAPE, MSE, RMSE etc. for my VAR forecast, but i simply cannot get it to work. ...
0
votes
0answers
15 views

Passive Combination of two Classifiers at the level of Class Labels

I have three classifiers - 1. Vision based classifier trained to detect class labels such as pedestrian and cars 2. Radar based classifier to detect same class labels 3. Lidar based classifiers to ...
0
votes
1answer
36 views

Model selection based on the closest fit to another model

For each subject I have two measures, let's say $X$ and $Y$, that correspond to the same measurement but each from a different measurement device. In theory we assume that ideally these two ...
2
votes
0answers
17 views

Why is pseudo-Bayesian Model Averaging using WAIC giving counterintuitive results for my model? Issue with unconstrained data uncertainty?

I'm currently working on a model which fits a pair of curves which are parameterized by some shape and scaling parameters. I want to produce weights for two different models. In my first model, which ...
0
votes
0answers
10 views

Bayesian Inference: Predicting Model Validity From Model Evidence

Bayes Theorem states that $$P[\theta | D] = \frac{P[D|\theta]P[\theta]}{P[D]}$$ where model evidence is defined as $$P[D] = \int P[D|\theta]P[\theta]d\theta$$ I have learned that model evidence ...
0
votes
0answers
8 views

What causes a chol.default(tmpvc) error in the vuong test when comparing non nested models?

I have a series of ordinal logistic regressions. Each predicts the same outcome (Y), and each has a single predictor (X1, X2 or X3 (All are strongly correlated)). I want to determine whether any of ...
0
votes
0answers
15 views

Compare two classifiers that works on two different datasets

I have 2 activity-recognition classifiers working on two different dataset (representing different repetitions of the same movement). Since the performance values are obtained from different samples, ...
0
votes
0answers
23 views

Is there an AIC and BIC equivalent for MAP?

I would like to know if there is an AIC or BIC equivalent for maximum a posterior estimation. I'm trying to compare several different models that have been fit using MAP, but I am unsure of the best ...
2
votes
1answer
70 views

Alternatives for Diebold-Mariano test when comparing the best forecast among many against a benchmark

Suppose I encounter a new forecasting method and I wish to evaluate it against a benchmark. I can obtain forecasts from the two methods and compare them to actual realizations and thus obtain the ...
1
vote
1answer
101 views

Dietterich's 5x2cv paired t-test for regression problems

In 1998 Dietterich proposed an algorithm for comparing classification algorithms using a series of cross validated t-tests: Dietterich, Thomas G. "Approximate statistical tests for comparing ...
1
vote
0answers
71 views

K-Fold Cross Validation, the right way

I searched for this in many forums, but didn't find any answers that would answer my question (or I didn't understand correctly). So, I will post here. I have a dataset A I have a machine learning ...
0
votes
1answer
30 views

Is the AIC of an ARIMA model comparable with the AIC of a regression with ARIMA errors?

I already know that models from different "families" cannot be compared through AIC (or other information criteria such as BIC or HQIC), however, I'm not sure about the specific case of ARIMA and ...
0
votes
0answers
30 views

compare coefficients from different regression

I estimate the following models using the Hausman-Taylor estimator: $$y_{i,t} = a_{0} + B_1 controls_{i,t} + \beta_1x_{i,t=2000} + B_2 Year_t + B_3 x_{i,t=2000}*Year_t + e_{i,t}, (1) $$ $$y_{i,...
0
votes
0answers
8 views

Questions Tags Users Unanswered Connection of the critical value for KS distance and model complexity

I asked a similar question is math exchange, but here it seems to fit better. I wonder about a relationship of the critical value for the KS distance (with Lilliefor correction) and model complexity? ...
0
votes
1answer
58 views

Nested Cross-Validation with Test Set (Model Selection)

I have seen many examples on Nested CV where someone takes the entire dataset and performs nested CV on it. My question is: for model comparison shouldn't we first split the original dataset into ...
2
votes
1answer
33 views

What are some of the most correct/accepted ways to tune and compare different models in an academic context?

Those days, I have been reviewing different academic papers which mainly compare the performance of different machine learning methods on a particular problem. And I was surprised by the variety of ...
0
votes
0answers
15 views

Bayesian Comparison of Dependent Correlation Coefficients

I would like to compare the correlation coefficients from two different models of the same data points. One of them is the raw bivariate data (Value vs Time), and the other is given a biologically-...
0
votes
1answer
58 views

Is it valid to use Anova (in R) to compare alternative multinomial log-linear models?

I am familiar with the idea of comparing alternative linear regression models using anova(model1,model2), for models fitted using ...
0
votes
0answers
24 views

Choosing the best aggregation location measure

I'm trying to analyze a granular dataset that is composed as follows: ...
2
votes
0answers
96 views

WAIC for model comparisons--overly conservative?

I'm having a hard time wrapping my head around the relationship between model posterior predictions and model comparisons via WAIC. Specifically, how do I interpret findings where a model including ...
4
votes
2answers
264 views

Should I cross-validate metrics that were not optimised?

I want to compare two models. Say I have two objective functions on the same data $f(X,y,\theta)$ and $g(X,y,\theta)$ that both evaluate the models performance in ways that I am interested in ($\theta$...
0
votes
0answers
12 views

How to compare our model with state-of-the-art deep models which are given for other frameworks?

I am preparing a paper presenting a new deep model. I wonder how to compare our model with the state-of-the-art models. In other papers, authors used the same dataset but not with same data split (...
1
vote
1answer
125 views

Can I use the AIC to compare ordered logistic regression models based on different datasets?

I performed ordered logistic regression using the polr() command in R. I did this with the same independent and dependent variable (the X and Y are the same for every model), but the exact data in ...
0
votes
0answers
10 views

Reporting difference in distributions for text

I am working on a problem that can be translated to determining which of two corpora generated a given sample text. I have a hypothesis for before some processing and other for after processing, and ...
1
vote
0answers
18 views

DIC to compare models with different numbers of parameter?

I am interested in comparing hierarchical Bayesian models based on the same dataset but differing in their spatial and temporal resolution. In short, I am looking at animal population changes over ...
1
vote
1answer
40 views

Comparing two machine learning models

Let's say I'm working with two models, ResNet-50 and ResNext-50, and I want to compare the results. Due to the stochastic nature of deep learning, would it be advantageous or even encouraged to seed ...
0
votes
0answers
22 views

Loss function for one-step-ahead volatility forcasts

I'm trying to perform the MCS test using the R-package "MCS" to compare GARCH-MIDAS Models. The loss function requires as inputs a vector with some realized volatility measure ˜ σt+1 (I chose the ...
0
votes
0answers
19 views

Difference between null and saturated log-linear models

I have data from an experiment testing the number of 'cases' at each of three measurement points (0, 12, and 24 weeks). I am interested in whether the proportion of cases across these measurement ...
0
votes
0answers
26 views

Likelihood ratio to quantify the similarity between one sample with two other matched samples

I conducted a study with 3 conditions and N subjects. All subjects performed all conditions once. I would like to know if the first condition is similar to the second or third condition. Formally, ...
5
votes
2answers
142 views

Bayesian methods are about averaging over uncertainty rather than optimization. Explain?

I came across the statement "The key ingredient in Bayesian methods is to average over your uncertain variables and parameters, rather than to optimize". Can someone explain why this is? ...
0
votes
2answers
140 views

Bayes factors and predictive accuracy in model comparison in rstan / brms

Despite reading up on the subject, I can't wrap my head round it, so the question remains on shaky grounds, and responses along the lines of "read chapter x" are very welcome. What I'm doing is I'm ...
1
vote
0answers
260 views

NLS model comparisons by ANOVA

I am comparing NLS models from two groups, males and females. I compared the models by ANOVAs, but I am hesitating which is the correct model comparison to use: a) Subgroup Males vs Subgroup Females ...
0
votes
0answers
10 views

Deterioration of the accuracy of a system over time

I have a system that compares the predicted variable with the true variable by calculating the absolute error percentages. . Where $\pi$ is the predicted variable and P is the true variable. And you ...
1
vote
2answers
35 views

How to compare two classifications, visually?

I have two classifiers, and I was asked (for learning purposes) to compare their predictions, visually. The target value is a real value (sentiment) What could be a good way to compare them?
0
votes
0answers
12 views

Marginal effect from two models with different dependent variables

This question is somehow related to this one and this one. I have two models with the same independent variables on the same dataset. Model1: Y1 = alpha0 + alpha1 X1 + alpha2 X2 + error1 Model2: ...
3
votes
2answers
120 views

Permutation test comparing nested non-linear models with an exchangeable dummy variable

This question is closely related to an earlier question, but I realized my case was actually a lot more specific than the way I formulated it there, in ways that I think merit a separate answer. I ...
0
votes
0answers
10 views

Model comparison with new IV over data subset: to rebuild base model or not

I'm modeling a few DVs with a set of IVs via glmer. I have successfully built a drop-1 nested model comparison framework to test the significance of each variable. ...
3
votes
1answer
346 views

Interpretation of R output from Cohen's Kappa

I have the following result from carrying out Cohen's kappa in R ...
0
votes
1answer
250 views

How to interpret/compare R2 score(s)? [duplicate]

I know that an R² score of 1 is a perfect fit of the model to the truth, a 0 is an constant output regardless of the input, and that negative values are possible when the output varies, but there is ...
1
vote
0answers
49 views

AIC for latent variable models

I'm trying to use BIC/AIC for model comparison and want to know what the number of parameters is. The models I'm unsure about are linear Gaussian state space models with nonlinear observations. ...
0
votes
1answer
52 views

Can I compare AIC values for similar models fitted for two samples from the same population?

If I have two samples (n=300) from the same population and I'm fitting a GLMM (Generalised Linear Mixed-effects Model) with a similar response and explanatory variables but with a completely different ...