Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

intepreting AIC and drop1 of models

Consider the following case: we have continuous response A, and indicator B,C,D. ...
0
votes
1answer
19 views

Train data selection for model comparison

Let I have data to compare two models. My first model wil be based on Arificial Neural Network(ANN) My second model will based on linear regression(LR) For the ANN, I have to divide data into 3 ...
0
votes
0answers
26 views

Classification model - metrics to optimise so that number of Predicted positives equals the number of Actual positives

I am working on a classification problem, predicting if people travel overseas or not in the future, and have been using cross-validation to tune a model. I'm trying to decide on the best metric to ...
0
votes
2answers
41 views

Rejecting a model in favor of another

Let's say currently there is a specific time series forecasting model A in use for some case and I want to test and prove whether another model B is significantly better and should replace model A. ...
1
vote
0answers
22 views

Comparing two multi-class machine learning classifiers using Stuart Maxwell Test

I need to compare 2 multi-class classifiers. So, to assess whether the difference between the two are statistically significant I have taken the following steps: obtain prediction on test data using ...
3
votes
1answer
30 views

Comparing the accuracy of binary classifiers using iterated cross-validation

Let's say that you want to compare two binary classifiers (e.g., LDA and linear SVM) for a given research question, the question being "which one will probably perform best for the problem at ...
1
vote
0answers
19 views

Bayesian comparison for two linear models with same response variable but different predictors?

I ran several separate linear models with the same response $y$, but with $n$ different predictors, i.e.: $y = \alpha + \beta x_0$, $y = \alpha + \beta x_1$, ..., $y = \alpha + \beta x_n$. All ...
0
votes
1answer
20 views

Comparability of classifier probabiliy estimates

Consider that you have 3 classification models ($model_1$, $model_2$ and $model_3$) that are designed to model whether or not customers are interested in different products of your company (binary ...
0
votes
1answer
15 views

How to compare two VAR time series models in R?

I have generated two VAR time series models in R for a dataset. My query is how can I compare those two models based on any kind of metrics like forecasting power :accuracy / f-1 score or something ...
0
votes
1answer
29 views

Comparing posterior predicted probabilities with “known” to be true probabilities

Following this description I have implemented a Bayesian logistic regression (BLR) model on some data. Lets say I have this kind of data: ...
4
votes
1answer
53 views

Is there a formal test for model comparison between polynomial regression and piecewise/segmented regression? (i.e., are these models nested?)

I am interested in comparing three models a linear regression model: mod_linear <- lm (dv ~ iv) a polynomial regression model: mod_polynomial <- lm (dv ~ iv + I(iv^2)) a one-breakpoint ...
1
vote
0answers
9 views

Which model to select of two similarly performing, models with similar architecture and number of parameters, but different depths

I am training U-Net models for two-class semantic segmentation (foreground/background). I have tested different depths of the U-Net along with different number of filters in the first conv-layer (the ...
0
votes
0answers
23 views

Comparing the marginal effect of a GLM to the OLS estimates

My question is, whether there is any way to (somewhat) compare the marginal effect of a GLM estimate to an OLS estimate. As in, "since the OLS and GLM results are very similar, I will favour OLS ...
0
votes
1answer
34 views

Comparing the marginal effects of glm output to polr output

I have a dependent variable that is technically ordinal, so I ran a ordered probit model (polr). However, an ordered probit model does not produce any residuals ...
0
votes
0answers
16 views

Validity of time series mean comparisons

I am looking to compare time series obtained from physiological responses to different treatments. I know that I should model these time series using common models, eg. AR, ARIMA, etc., and then ...
0
votes
0answers
8 views

Is it legitimate to compare likelihood ratios from different datasets?

I have two nonlinear models, $M_1$ and $M_2$, where $M_2$ has all parameters of $M_1$ and a few additional ones. Since $M_2$ is more expressive than $M_1$, it will always be at least as good as $M_1$ ...
0
votes
2answers
69 views

How to prove statistical significance, python

I'm new in statistics and would be so grateful if you give me some insights: I have two big tables - results of work Model_1 and Model_2. I created and calculated statistics - something like Precision ...
1
vote
1answer
36 views

Statistical test for comparing performance metrics of two regression models on a *single* validation dataset?

What is an appropriate way to apply a statistical hypothesis test for evaluating model performance metrics (e.g. MAE, MAPE, MSE) between two regression models based on a single holdout (validation) ...
0
votes
1answer
47 views

what is a good measure of goodness of fit for survival models that can be used for comparison between models?

I don't see any of such measure in the output from coxph() in R (Cox proportional Hazard model). Is there a goodness of fit measure for survival models similar to R2 for linear regression? Update: ...
2
votes
2answers
69 views

Which statistical test to compare same model with different parameters?

I have two datasets on people buying apples based on weight and price. One dataset in 2019 the other in 2020. I estimate a logit model with Utility = betaWeight * weight + betaPrice * price. Training ...
0
votes
0answers
22 views

Application of BIC when using data to estimate hyperparameters

I plan on using the following equation to calculate the Bayesian information criterion of various linear models applied to the same dataset: $n\ln(RSS/n) + k\ln(n)$ (I believe this equation applies ...
0
votes
0answers
17 views

Can you use information criterion to decide if random effects are important in your model?

I want to know if adding random effects in a model improves its predictive performance. I have a model with fixed effects below: m1<- stan_glmer(a~b+c) Which I want to compare with a mixed effects ...
0
votes
1answer
23 views

Searching papers that compare different GARCH models

Like the title says, I'm searching for papers that compare the performance of different GARCH models (mainly the standard GARCH, EGARCH and GJR-GARCH). I'm sure there are some standard papers that ...
0
votes
0answers
29 views

What statistical test should I use for this analysis where I want to compare the similarity of two histograms and return a single value percentage?

I need some help with what statistical test I should use for my data analysis. I have a Data set A, which is an array of 5000 numbers, ALL of which are zero, and a Data set B, which is an array of ...
0
votes
0answers
13 views

Is a classification comparison between Linear Discriminant Analysis and a SVM trained on said linear discriminants appropriate?

I am currently trying to investigate the classification accuracy of two models on a wide dataset (79*222), with 4 balanced classes. The models are: Principal Component Analysis, Linear Discriminant ...
1
vote
0answers
30 views

Do conventional thresholds for global fit indices (e.g. AIC) hold for models based on very large data sets?

Problem/Question in short: I have estimated 5 generalized linear mixed models and subsequently compared their levels of relative fit according to AIC. These models are based on a very large dataset of ...
0
votes
0answers
5 views

Compare evidence of models with noise at different levels of a hierarchy

I have a data set with stimuli $X$ and responses $Z$. I want to compare two mechanistic models $A$ and $B$ trying to explain the relationship between $X$ and $Z$ Both models assume two hierarchical ...
2
votes
1answer
118 views

Mean absolute percentage error with respect to predictions

A friend of mine has suggested that instead of using mean absolute percentage error, i.e. $$ \frac{1}{N}\sum_{i=0}^N \left| \frac{A_i - P_i}{A_i} \right| $$ (where $A_i$ denotes an actual value, $P_i$ ...
3
votes
1answer
55 views

SVMs are outdated for time series classification - Citation needed

I'm looking for a strong publication I can cite to prove that SVMs perform worse for time series classification tasks in comparison to other methods (such as ANNs, Decision Trees, Gradient Boosting ...
1
vote
1answer
25 views

Why does the first term in this marginal likelihood approximation decrease with model complexity

In Bishop's PRML chapter 3, he presents the following approximation for the marginal likelihood in equation 3.72 $\ln P(D) \approx \ln P(D| \boldsymbol{w}_{map}) + \ln M\frac{\Delta w_{posterior}}{\...
0
votes
0answers
34 views

Expanding the 5×2 cross-validation with a Modified Paired Student t-test

I have run multivariate linear regression on three datasets, which share a common output (only the input features differ). These three datasets are paired (e.g. observation 1 in the first dataset is ...
0
votes
1answer
27 views

Check if a variable has significantly different effects in 2 samples

I have a variable (Y) measured on 2 different samples (X, 0=clinical, 1=control). I verified that a third variable (BMI, 3 classes) has an interaction with Y based on point plots by plotting X,Y for ...
0
votes
0answers
10 views

mixed effects models when variable with missing values depends on the value of another variable

this question is very similar to Missing values in a variable depending on the values of another variable, but my problem is a bit more complicated. I have the following panel where I follow the ...
1
vote
0answers
27 views

In ARIMA models, given identical performance and same number of total parameters, is a pure AR model preferred over an ARMA model?

Say ARMA(5,0) and ARMA(3,2) provide the same (and best) cross-validation results. Is there any sense in which we can appeal to the principle of parsimony to argue in favour of ARMA(5,0)?
0
votes
0answers
13 views

Reduced chi-squared for models fit to different subsets of data

I have some data to which I am fitting piecewise linear models. I want to select different subsets of the data, fit a model to each of them, and then compare which subsets are best able to be ...
0
votes
0answers
27 views

Appropriate hypothesis test for comparing 3+ regression models

I'm struggling to locate the exact type of hypothesis test I need. Here is the situation. I have field plots (n > 30,000) which have been established to measure forest attributes. The response of ...
1
vote
0answers
227 views

Differences between time-series regression versus cross-sectional regression

I am currently trying to grasp the differences between time series regression and cross-sectional regression, because these terms are often used in papers about market predictability. I understand ...
0
votes
0answers
37 views

Likelihood Ratio Test for model selection

I have a dataset with 6 variables, a1, a2, a3, a4, a5, a6 the outcome is Y. This is the model fit statistics after including only first three variables , a1,a2, a3 ...
1
vote
1answer
54 views

Comparing model efficiency

I hope you all don't mind me asking this question. I have two models : general linear mixed effects model ...
3
votes
1answer
36 views

Which metric to use to evaluate highly imbalance classification model performance

I have to do classification model to predict the possibilities of person getting cancer based on certain attributes. The data is highly imbalanced. As per client requirement I have to report model ...
1
vote
0answers
73 views

Use mean squared error (MSE) for comparing model fits of Bayesian models

I want to use mean squared error (MSE) to assess/copmare the model fit of the Bayesian models. The formula for MSE is $MSE=\frac{1}{n}\sum^n_i{(y_i-\hat{y}_i)^2}$ I'm not sure how MSE is used for ...
0
votes
0answers
11 views

Is there a different name for a non-asymptotic chi-square difference test or is it always a Likelihood Ratio Test?

I already know that $-2log(Likelihood Ratio)$ is asymptotically $\chi^2$ distributed according to Wilks' theorem. It seems that a comparison of nested models involving computing the difference of $\...
0
votes
0answers
13 views

Covariance between parameters from different regression models

Is there a formula for the covariance between regression slopes from different models, fitted to the same data? For example, if I have a finite and fixed sample, $S$, and models: $Y = b_1X$ $Y = b_2X ...
0
votes
0answers
49 views

Choice of Time Series Models for Forecasting Data

I have a question that to most readers here probably think is fairly simple to answer, but never the less, out of curiosity I'd like to ask: Given a time series, yt, which is stationary (say the log ...
2
votes
2answers
121 views

AIC calculation with very low negative log likelihood

I am using AIC formula (AIC=2k−2lnL) to compare different exponential models. I know that this formula is used to penalize complexed models (with high number of parameters). The problem I have is that ...
0
votes
0answers
14 views

Post-hoc analysis of neural network predictions

I have trained a model in PyTorch. My model predicts results of football games. I hypothesize that certain games in my test-set will have higher accuracy. One of the variables would be the start of ...
0
votes
1answer
71 views

AIC, pseudo-R2, or log likelihood to compare models?

I am comparing the effect of climate, across three different time brackets, on a variable. I am interested in choosing the model that best predicts the variable to answer across which timescale the ...
7
votes
0answers
209 views

How to calculate percent partial deviance explained by each predictor variable in a GAM model?

I am trying to find a sensible way to calculate the deviance explained by each predictor variable in a GAM model and need some input on my calculations. Following Simon Wood's example on the thread ...
0
votes
0answers
11 views

Does DIC favor more complicated model?

I'm doing a model selection/comparison based on two criterions - WAIC and DIC. When I consider the WAICs, my model has the smallest WAIC. However, DIC of my model is slightly bigger than DICs of few ...
1
vote
0answers
57 views

ARIMA vs. ADL model

I have estimated an MA(3) model and an ADL model on the differenced US unemployment rate. However, I'm in doubt as how to compare the two models? Does it make sense to compare them using information ...

1
2 3 4 5
10