Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

Filter by
Sorted by
Tagged with
0 votes
0 answers
12 views

Bayesian Model selection vs Model comparison

Is anyone aware of any articles or book chapters about the distinction between model selection and model comparison in bayesian multilevel modeling? I am fitting several competing growth models using ...
0 votes
0 answers
23 views

How to statistically determine if two files have identical content?

I'm writing a duplicate file finder application, and I do not want to have to read every byte of every file to determine which files are duplicates. Hard drive storage is segmented into clusters of X ...
  • 101
1 vote
1 answer
29 views

How to compare variance explained in two regression models?

Let's say I have a measurement of height of 200 individuals as $y_1$ and fit a simple linear regression model as: $y_1 = x_1 + x_2 + \epsilon_1$ where $x_1$ and $x_2$ are some subject metadata I ...
  • 231
0 votes
0 answers
6 views

Choosing between univariate GLMMs to assess inclusion in a multivariate model

I am currently doing a project on environmental determinants of malaria vector distributions. I'm using remote sensing data for environmental variables linked via GIS. I have run univariate (binomial) ...
0 votes
0 answers
11 views

How would you compare competing Bayesian hierarchical models that are estimating the same time-series?

Suppose we have two models $\mathcal{M}_1$ and $\mathcal{M}_2$ that attempt to estimate a time-series $\boldsymbol{\theta}$. Let's assume that I actually know the true values for $\boldsymbol{\theta}$....
  • 1,615
3 votes
2 answers
100 views

Compare GLM AICs with different likelihoods?

If I have a generalized linear model (GLM) with a particular likelihood, and I have another GLM of the same data (say nested within the first model), I can compare the model performance using Akaike ...
  • 35.7k
4 votes
2 answers
262 views

Valid model comparison/selection? Poisson, negative binomial, zero-inflated poisson, z-i negative binomial with AIC

I want to model counts of an event in a pre-post design. A sample dataset could look like this: ...
0 votes
1 answer
9 views

Which factor should I consider to choose the best model, stationarity or lowest MSE?

Recently, I did a project where I modelled a dataset of Gold price. I used ARIMA(1,1,1) and ARIMA(2,2,3) to model the data. The results that I get was ARIMA(1,1,1) was not stationary but the MSE value ...
0 votes
1 answer
58 views

$R^2$ model comparison in test data

I want to compare two models of the form: ...
1 vote
1 answer
28 views

Independent Samples T-test vs Paired Samples T-test for hypothesis testing (machine learning algorithms comparison)

The scenario is as follows: I have 1 dataset and I want to compare the performance of different algorithms. Let's say A, B, C. I train each algorithm a 1000 times in one training loop. At the ...
  • 11
1 vote
0 answers
26 views

Specification of regression model based on two data sources

Suppose that I have two different datasets from surveys in which the dependent variable is measured on different scales and I want to evaluate the effect the survey method (i.e. the measuring scale) ...
  • 11
1 vote
0 answers
4 views

Rank Correlation for Paired Predictor to compate 2 GLM models: appropiate objects and p-value

Dear Professor Harrell and users, Thank you for this post: How to assess the incremental additive information of a new predictor in a Cox model using R?, it has been really helpful for my work. I am ...
2 votes
2 answers
60 views

Is there a metric to compare machine learning models which takes into account how many trainable parameters they have?

Assume we have two models for the same task. One with 1K parameters and one with 1M and they achieve accuracy as follows Model Params Accuracy small 1 000 0.8 big 1 000 000 0.9 I would like to ...
0 votes
0 answers
71 views

SEM model comparison

I have problem choosing a better measurement model between two (i.e., A and B). Both models have acceptable fit indices. For example, the RMSEA of A is .056; the RMSEA of B is .067. By RMSEA, A seems ...
1 vote
2 answers
138 views

Exploring Alternative Pythagorean Expectation Formulae

In a regular match, a team score $a_{i}$ goals and allow $b_{j}$ goals, both are natural numbers). The result will depend on: $$\left | a- b \right |\left\{\begin{matrix} \geq e. 2.0\,{\rm points}\,{\...
0 votes
0 answers
18 views

anova for model fit return NA

So, i have a continuous outcome and two binary predictors. I want to compare different models between them ...
1 vote
1 answer
96 views

Significance test for comparing different 10-fold cross-validated Machine Learning Regressions

Is there a recommended significance test for comparing different 10-fold cross validated regressions? For instance, I want to compare the performance of LASSO against Random Forest for my dataset. ...
0 votes
0 answers
21 views

Statistical test for forecast performance over multiple runs

Lets say I have a time series, create a training and test set, and I want to compare the predictive accuracy of two models, by measuring e.g. the mean absolute error (MAE) over the test set. I know ...
0 votes
0 answers
26 views

How to interpret the output of compare_models in R?

I have been trying caret library, specially the part to compare two or more obtained models. Some of the articles that I read use the compare_models function. As I see on the documentation, it runs a ...
1 vote
0 answers
27 views

How to compare different models in their ability to forecast the value-at-risk with Diebold-Mariano test?

I made value-at-risk forecasts for different models for the 95, 97.5 and 99%. I also made a dummy which equals 1 if the true return was below the value-at-risk, 0 otherwise. How can I compare those ...
  • 41
0 votes
0 answers
309 views

An explanation of swap set analysis and swap set table

I'm trying to understand swap set analysis and the way results from such an analysis are presented. This analysis is a way to compare two models to determine which one is better. The results are ...
  • 141
0 votes
0 answers
21 views

Comparing multiple glm models with different y-variable for best fit

I have seven different GLM models. Let's say: ...
7 votes
3 answers
286 views

Test the hypothesis that the performances of k machine learning models on the same test set is the same

I have $k$ Machine Learning models, trained on the same training set, and I want to test the hypothesis that their performance on a fresh test set is the same. See EDIT 1 below for what I mean with ...
  • 16.2k
2 votes
1 answer
38 views

Can I compare the same Linear Regression Model over time?

I'm performing a linear regression model with respect to voter participation for one of my class papers and I was curious if I could apply the same model to various time periods and then analyze/...
0 votes
0 answers
48 views

How to compare distributions with errors on the data points?

Here's a mock set-up of my problem: I have two non-normal probability density distributions (PDFs), $A$ and $B$. Distribution $A$ has error measurements for each data point while distribution $B$ ...
0 votes
0 answers
144 views

Is correct to apply Wilcoxon Signed-Rank Test on multiple datasets?

I am learning about statistical tests and trying to apply this concept in machine learning by comparing different classifiers in multiple datasets. As for my understanding, to compare two models in ...
0 votes
0 answers
9 views

Fairness of data-driven comparison: LR vs. mixed effects LR

In terms of comparability of LR vs. mixed effects LR, I wonder if it is a problem since number of variables is large (so far 55). However, the RF model is modelled using 55 variables, but the mixed ...
0 votes
0 answers
30 views

R: How to determine lmer baseline in k-fold cross validation

For testing whether a predictor improves a linear mixed-effects model fitted with lme4, I used to fit a baseline model and the baseline model with an additional ...
  • 33
0 votes
0 answers
18 views

model to explore correlation of pollution long-term exposure with genetic mutation rates

I have pollution data of several US cities, as example NY, Boston, and Chicago. I need to ...
  • 197
0 votes
0 answers
30 views

Can we compare whether different groupings of data improve model accuracy?

I have data from 100 different lab incubations of manure samples. For each sample, a 3-day incubation was done, measuring values (y-axis) against time (x-axis). I want to perform a linear regression ...
1 vote
0 answers
40 views

Is there a statistical test for matrix symmetry? [closed]

I have collected some data and done some processing until the question I'm faced with is - "is the data matrix $X$ a symmetric matrix?". Note that elements of $X$ represents event counts. ...
  • 121
0 votes
0 answers
25 views

How to compare different LSTM models with different features?

I have 3 LSTM models with different features. LSTM1 is trained using Adjusted Close LSTM2 is trained using Adjusted Close and <...
0 votes
0 answers
31 views

Comparing models to pick the best one

I am trying to compare these 5 models but am unsure of which is best. Model 1 has the worst fit and model 5 has a p value above 0.05 so i have ruled these out. Im not sure if i should go on R-squared ...
  • 1
0 votes
0 answers
59 views

What is the number of free parameters in an n-component GMM?

I am trying to calculate BIC = -2logL + log(N)d where d is the number of free parameters or degrees of freedom. If I am fitting guassian mixture model to the data, ...
  • 1
1 vote
1 answer
24 views

How to compare nested multilevel path models

I have two multilevel path models, one is nested in the other and I want to compare the models to see whether I should prefer the full or nested model. The models have cross-level interactions and I ...
  • 13
1 vote
0 answers
40 views

Formal comparison of two mediation effects

Is there a method to statistically compare two mediation effects, given the same IV and DV? We have two different mediators and we want to compare the resulting models, as in: Model 1: $X – M_{1} – Y$ ...
  • 333
1 vote
0 answers
23 views

Model comparison: what to do when reduced model doesn't converge?

This is a general question about how to do hypothesis testing via model comparison: I want to test the significance of several different predictors in a data set using model comparison. After a lot of ...
  • 31
1 vote
0 answers
64 views

Comparing effect size for regression coefficients for models with different dependent variables

I want to compare the regression coefficients $\beta_{1}$ and $\beta_{3}$ in the two models which have the same independent variables but slightly different dependent variables: $$ \Delta{y}_{1}=\...
1 vote
0 answers
19 views

Comparing models with partially observed data and limited supervision

Setup: Suppose we have $n$ questions and for each question $m$ answers. For each question, a model, Model A, selects one answer out of $m$. Note that for each question, zero, one, or more answers ...
  • 670
2 votes
0 answers
25 views

Model selection: comparing Bayesian models with likelihood vs likelihood-free (Approximate Bayesian Computation)

I have two families of models that can possibly explain the data at hand. One family is rather process-based, using fairly complicated simulations and Approximate Bayesian Computation to estimate the ...
  • 121
1 vote
1 answer
60 views

GARCH model sensitivity to distribution assumptions

I am trying to fit an ARMA(4,4)- GARCH(1,1) model to return data, where the distribution of returns is highly leptokurtic. I plan to see whether autocorrelations exist in the data even after ...
0 votes
0 answers
23 views

Comparing model fit on data aggregated at different levels

I'm fitting a non-linear model to data aggregated at 5 different levels and want to deduce at which level the model fits the data best, i.e. compare the model fits at the 5 different aggregation ...
0 votes
0 answers
22 views

Is it okay to use AIC values generated from summary() for glmer?

I am conducting a random effect logistic regression using glmer in R. I have 13 different predictor variables, which I am evaluating at 4 different spatial scales (i.e. I have 13 variables derived ...
0 votes
0 answers
82 views

Why does a Decision Tree algorithm outperform the Random Forest Algorithm in certain cases?

Currently I write my master thesis that deals with the binary prediction of university dropouts (dropout - yes/no). In the thesis, I compare the performance of three different classification ...
0 votes
0 answers
21 views

Time Series Forecasting with Different Time Horizon for Comparing Models?

At the moment I am dealing with a time series problem. The data I have is about 6 years and in daily frequency. I want to try out different models on the data and I came up with an experiment: ...
0 votes
0 answers
20 views

Combination of multiple datasets to compare models with experiments

I work on metal fatigue lifetime estimation of material under different loading conditions. I have performed a series of experiments on a certain material to determine the lifetime of the material for ...
  • 1
0 votes
0 answers
28 views

what is the difference between RMSE and Mc error (in Bayesian estimation)?

Please, I want to know if there is a difference between Root mean square error (RMSE) and Monte carlo error (Mc error). I need to compare my models but in Bayesian modelling I found a value like Mc ...
1 vote
0 answers
23 views

Is there a consensus on bad practice when doing model comparison in deep learning?

Lately I've been thinking about how to maintain the Equity property when Benchmarking DL for research. On Wikipedia, Equity property is defined as "All systems should be fairly compared." ...
0 votes
0 answers
31 views

Null model comparisons with mixed models

I have a question about the general practice of null model comparisons. I understand why they can be useful in assessing model quality for prediction. My question comes when involving varying (random) ...
  • 51
1 vote
0 answers
55 views

Goodness of fit of two models

My two models that I would like to compare are Weibull distribution and Quantile regression (in the case of Weibull distribution I have estimated the quantile curves). I would like to know if there is ...
  • 161

1
2 3 4 5
11