Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

Filter by
Sorted by
Tagged with
1
vote
0answers
16 views

Do conventional thresholds for global fit indices (e.g. AIC) hold for models based on very large data sets?

Problem/Question in short: I have estimated 5 generalized linear mixed models and subsequently compared their levels of relative fit according to AIC. These models are based on a very large dataset of ...
0
votes
0answers
4 views

Compare evidence of models with noise at different levels of a hierarchy

I have a data set with stimuli $X$ and responses $Z$. I want to compare two mechanistic models $A$ and $B$ trying to explain the relationship between $X$ and $Z$ Both models assume two hierarchical ...
2
votes
1answer
27 views

Mean absolute percentage error with respect to predictions

A friend of mine has suggested that instead of using mean absolute percentage error, i.e. $$ \frac{1}{N}\sum_{i=0}^N \left| \frac{A_i - P_i}{A_i} \right| $$ (where $A_i$ denotes an actual value, $P_i$ ...
3
votes
1answer
39 views

SVMs are outdated for time series classification - Citation needed

I'm looking for a strong publication I can cite to prove that SVMs perform worse for time series classification tasks in comparison to other methods (such as ANNs, Decision Trees, Gradient Boosting ...
0
votes
1answer
17 views

Why does the first term in this marginal likelihood approximation decrease with model complexity

In Bishop's PRML chapter 3, he presents the following approximation for the marginal likelihood in equation 3.72 $\ln P(D) \approx \ln P(D| \boldsymbol{w}_{map}) + \ln M\frac{\Delta w_{posterior}}{\...
0
votes
0answers
8 views

Expanding the 5×2 cross-validation with a Modified Paired Student t-test

I have run multivariate linear regression on three datasets, which share a common output (only the input features differ). These three datasets are paired (e.g. observation 1 in the first dataset is ...
0
votes
1answer
25 views

Check if a variable has significantly different effects in 2 samples

I have a variable (Y) measured on 2 different samples (X, 0=clinical, 1=control). I verified that a third variable (BMI, 3 classes) has an interaction with Y based on point plots by plotting X,Y for ...
0
votes
0answers
8 views

mixed effects models when variable with missing values depends on the value of another variable

this question is very similar to Missing values in a variable depending on the values of another variable, but my problem is a bit more complicated. I have the following panel where I follow the ...
1
vote
0answers
27 views

In ARIMA models, given identical performance and same number of total parameters, is a pure AR model preferred over an ARMA model?

Say ARMA(5,0) and ARMA(3,2) provide the same (and best) cross-validation results. Is there any sense in which we can appeal to the principle of parsimony to argue in favour of ARMA(5,0)?
0
votes
0answers
12 views

Reduced chi-squared for models fit to different subsets of data

I have some data to which I am fitting piecewise linear models. I want to select different subsets of the data, fit a model to each of them, and then compare which subsets are best able to be ...
0
votes
0answers
26 views

Appropriate hypothesis test for comparing 3+ regression models

I'm struggling to locate the exact type of hypothesis test I need. Here is the situation. I have field plots (n > 30,000) which have been established to measure forest attributes. The response of ...
1
vote
0answers
45 views

Differences between time-series regression versus cross-sectional regression

I am currently trying to grasp the differences between time series regression and cross-sectional regression, because these terms are often used in papers about market predictability. I understand ...
0
votes
0answers
35 views

Likelihood Ratio Test for model selection

I have a dataset with 6 variables, a1, a2, a3, a4, a5, a6 the outcome is Y. This is the model fit statistics after including only first three variables , a1,a2, a3 ...
1
vote
1answer
54 views

Comparing model efficiency

I hope you all don't mind me asking this question. I have two models : general linear mixed effects model ...
2
votes
1answer
28 views

Which metric to use to evaluate highly imbalance classification model performance

I have to do classification model to predict the possibilities of person getting cancer based on certain attributes. The data is highly imbalanced. As per client requirement I have to report model ...
1
vote
0answers
44 views

Use mean squared error (MSE) for comparing model fits of Bayesian models

I want to use mean squared error (MSE) to assess/copmare the model fit of the Bayesian models. The formula for MSE is $MSE=\frac{1}{n}\sum^n_i{(y_i-\hat{y}_i)^2}$ I'm not sure how MSE is used for ...
0
votes
0answers
9 views

Is there a different name for a non-asymptotic chi-square difference test or is it always a Likelihood Ratio Test?

I already know that $-2log(Likelihood Ratio)$ is asymptotically $\chi^2$ distributed according to Wilks' theorem. It seems that a comparison of nested models involving computing the difference of $\...
0
votes
0answers
11 views

Covariance between parameters from different regression models

Is there a formula for the covariance between regression slopes from different models, fitted to the same data? For example, if I have a finite and fixed sample, $S$, and models: $Y = b_1X$ $Y = b_2X ...
0
votes
0answers
47 views

Choice of Time Series Models for Forecasting Data

I have a question that to most readers here probably think is fairly simple to answer, but never the less, out of curiosity I'd like to ask: Given a time series, yt, which is stationary (say the log ...
1
vote
2answers
51 views

AIC calculation with very low negative log likelihood

I am using AIC formula (AIC=2k−2lnL) to compare different exponential models. I know that this formula is used to penalize complexed models (with high number of parameters). The problem I have is that ...
0
votes
0answers
11 views

Post-hoc analysis of neural network predictions

I have trained a model in PyTorch. My model predicts results of football games. I hypothesize that certain games in my test-set will have higher accuracy. One of the variables would be the start of ...
0
votes
1answer
30 views

AIC, pseudo-R2, or log likelihood to compare models?

I am comparing the effect of climate, across three different time brackets, on a variable. I am interested in choosing the model that best predicts the variable to answer across which timescale the ...
7
votes
0answers
140 views

How to calculate percent partial deviance explained by each predictor variable in a GAM model?

I am trying to find a sensible way to calculate the deviance explained by each predictor variable in a GAM model and need some input on my calculations. Following Simon Wood's example on the thread ...
0
votes
0answers
9 views

Does DIC favor more complicated model?

I'm doing a model selection/comparison based on two criterions - WAIC and DIC. When I consider the WAICs, my model has the smallest WAIC. However, DIC of my model is slightly bigger than DICs of few ...
1
vote
0answers
44 views

ARIMA vs. ADL model

I have estimated an MA(3) model and an ADL model on the differenced US unemployment rate. However, I'm in doubt as how to compare the two models? Does it make sense to compare them using information ...
0
votes
0answers
13 views

Metric to compare machine learning models with varying training data and feature set sizes

I have four machine learning models that were trained using different training sets and features. They were individually validated using the same test data and predictions were obtained. \begin{array} ...
1
vote
1answer
15 views

Compare fit of mixed model for different outcomes?

In the current literature, people are predicting X. I would like to show that the current models are more suited to predict Y than X. I have two equivalent models for the two different criteria. They ...
2
votes
1answer
34 views

The ratio of two Bayes factors for two opposite one-tailed hypotheses

I am trying to understand how Bayesian inference works, so this might be a very simple question. I have an experiment where I test two hypotheses predicting opposite results. Let’s say, hypothesis 1 (...
2
votes
1answer
27 views

Model Comparison with Regression

For a project I have to test a mediation model. I will do this with regression in R. The steps I take are the regular steps: Test if the IV has a relation with the DV (Regression) Test if the IV has ...
0
votes
0answers
38 views

random effect with one observation per group improves AIC drastically — explain

Now that I've posted an example (for a different purpose) at How will random effects with only 1 observation affect a generalized linear mixed model? (resulting in creating an account here, and being ...
0
votes
0answers
26 views

Checking for consistency and selecting the best among multiple measurement techniques

I am tasked with comparing $N$ statistical analysis methods (where $N$ is relatively small, say $N=5$), possibly not independent, for estimating some quantities $X,Y,Z$ based on some given data. Each ...
0
votes
0answers
14 views

Model Selestion - Compare AIC of data and its subset

I am trying to use AIC to assess the difference between replicates. In my experiment, each treatment has three replicates. The data from each replicates was fitted exponential model. And then combine ...
0
votes
0answers
13 views

Ranking of models on dev set does not match ranking on test

Using my train/dev/test split, I optimized four different model's hyper parameters over the development set. Suppose the ranking based on F1 score on the development set was: model 1 model 2 model 3 ...
1
vote
0answers
23 views

Can I use a (paired) t-test to compare the accuracies of classification models on a hold out test set?

When comparing the Brier scores or logloss/cross-entropy for multiple models on a hold-out test set, how can I be sure the 'best' model doesn't just have the lowest loss due to chance? When can I be ...
0
votes
0answers
32 views

Using model comparison to deal with collinearity in linear mixed models

I am working on a set given to me, which involves fitting logistic models with a couple of predictors, some of which are nearly perfectly correlated (.9), imagine, face features deviations of specific ...
1
vote
0answers
41 views

Regression model comparison/validation in an independent cohort: other method than c-index

What I have: I developed a logistic regression model (M1) to predict lymph node metastasis in cancer patients using the variables A + B + C + D + E and a training/validation data set D1, E is the new ...
3
votes
1answer
53 views

How to deal with missing values of explanatory variables when comparing models

I want to compare several logistic regression models. The different models are built using the same initial dataset. The models differ with respect to the explanatory variables included. Many of the ...
0
votes
0answers
18 views

Model selection for panel data

I was wondering about methods that could be used to compare different model specifications when dealing with panel data. In the case of cross-sectional data, a few applicable methods would be (in my ...
0
votes
0answers
62 views

To compare two KL-divergence scores, does the prior model have to be the same for both?

The KL-divergence compares a theoretical model $p$'s distribution with the empirical model $q$'s distribution, giving a score of $0$ if they, or their information contents, are identical. Say we have ...
0
votes
0answers
30 views

How to compare multiple forecasted time series data derived from an API against actual data from an official weather station?

Pardon, I am a novice of time series analysis. I have merged together meteorological data from two different sources, one derived from the Brazilian National Institute of Meteorology (INMET) and the ...
0
votes
0answers
45 views

Significance Test for Comparing RMSE values

I want to use the RMSE measurement to compare 3 statistical regression models against a baseline model. I want to do this based on a null hypothesis testing framework where I train the models using n-...
3
votes
1answer
55 views

Can I compare lmer models with different fixed effects using anova

I know that this question sounds familiar to some other, but I believe the responses were not clear in those and were focused on REML models. I would like to know if it is sensible to compare 2 or ...
0
votes
0answers
7 views

comparing calibration across two models

I have two models that predict a binary outcome. The range of model A is [0, 0.2] while the range of model B is [0, 1] with very sparse high probability predictions. Using the typical decile binning ...
0
votes
0answers
20 views

AIC model-comparison from one level to the next

Validated community, I have an fMRI dataset that consists of 35 subjects in which I measure a signal at 450 time points at around 3000 locations (voxels) in two hemispheres. I have two competing ...
0
votes
0answers
11 views

Using forecast errors to cross-sectionally compare quality of two different data sources (using sLDA)

At a high level conceptually, I have two different data sources for panel data and I am trying to compare the quality of each data source by comparing forecast errors from models predicting an outcome ...
0
votes
0answers
36 views

Comparing linear models, when one data set is a subset of the other

There are some questions about this site on the related question of how to compare models when the datasets are completely different. For example this question, or this other question, for which only ...
0
votes
2answers
241 views

ANOVA model comparison in R: AIC, BIC, LogLik vs RSS

When comparing 2 linear models in R with anova(mod1, mod2), I used to get a nice output showing AIC, BIC, LogLik etc: ...
0
votes
0answers
10 views

getting same AIC (or any other comparison criterion values) even after using different var-cov structures when comparing GLMM models

We are comparing models that are GLMM , in which for each one of them the fixed effects are exactly the same, but in the random effects portion, we used different variance-covariance structures (i.e ...
3
votes
1answer
62 views

R lmer model: add factors or reduce factors

In mixed effect models, do you add factors one by one? Or do you reduce the factors one by one? What I am doing is as follows. Are there any problems with the steps? Build a full model: ...
1
vote
1answer
81 views

Differences between formulas for AIC and BIC

I have a question regarding the information criteria AIC and BIC: I found different formulas for the AIC/BIC, the common ones including the likelihood $\mathcal{L}$ are $$AIC = 2K - 2 ln(\mathcal{L})\...

1
2 3 4 5
10