Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

Filter by
Sorted by
Tagged with
1 vote
0 answers
39 views

From inputs, updating $f\left(X_{i},Y_{i}\right)$ and $g\left(X_{i},Y_{i}\right)$ much other than $\overline{X_{i}}$ and $\overline{Y_{i}}$ to compare [closed]

(Premier League $2019$/$20$). The season was affected by the COVID–$19$ Pandemic while each team had a quarter of their schedule left (I call it $1/4$ because $1/2$ teams had $4$ away matches, and $5$ ...
user avatar
1 vote
1 answer
53 views

Significance test for comparing different 10-fold cross-validated Machine Learning Regressions

Is there a recommended significance test for comparing different 10-fold cross validated regressions? For instance, I want to compare the performance of LASSO against Random Forest for my dataset. ...
user avatar
0 votes
0 answers
15 views

How to conduct t-test for comparing the accuracy of two binary classifiers? [closed]

I am using two binary classifiers that predict the accuracy of samples over a dataset. I need to check if the difference in the mean accuracy between the two models is statistically significant. ...
user avatar
0 votes
0 answers
17 views

Statistical test for forecast performance over multiple runs

Lets say I have a time series, create a training and test set, and I want to compare the predictive accuracy of two models, by measuring e.g. the mean absolute error (MAE) over the test set. I know ...
user avatar
0 votes
0 answers
11 views

How to interpret the output of compare_models in R?

I have been trying caret library, specially the part to compare two or more obtained models. Some of the articles that I read use the compare_models function. As I see on the documentation, it runs a ...
user avatar
1 vote
0 answers
17 views

How to compare different models in their ability to forecast the value-at-risk with Diebold-Mariano test?

I made value-at-risk forecasts for different models for the 95, 97.5 and 99%. I also made a dummy which equals 1 if the true return was below the value-at-risk, 0 otherwise. How can I compare those ...
user avatar
  • 41
0 votes
0 answers
18 views

An explanation of swap set analysis and swap set table

I'm trying to understand swap set analysis and the way results from such an analysis are presented. This analysis is a way to compare two models to determine which one is better. The results are ...
user avatar
  • 141
0 votes
0 answers
12 views

Comparing multiple glm models with different y-variable for best fit

I have seven different GLM models. Let's say: ...
user avatar
7 votes
3 answers
203 views

Test the hypothesis that the performances of k machine learning models on the same test set is the same

I have $k$ Machine Learning models, trained on the same training set, and I want to test the hypothesis that their performance on a fresh test set is the same. See EDIT 1 below for what I mean with ...
user avatar
  • 15.9k
2 votes
1 answer
33 views

Can I compare the same Linear Regression Model over time?

I'm performing a linear regression model with respect to voter participation for one of my class papers and I was curious if I could apply the same model to various time periods and then analyze/...
user avatar
0 votes
0 answers
43 views

How to compare distributions with errors on the data points?

Here's a mock set-up of my problem: I have two non-normal probability density distributions (PDFs), $A$ and $B$. Distribution $A$ has error measurements for each data point while distribution $B$ ...
user avatar
0 votes
0 answers
43 views

Is correct to apply Wilcoxon Signed-Rank Test on multiple datasets?

I am learning about statistical tests and trying to apply this concept in machine learning by comparing different classifiers in multiple datasets. As for my understanding, to compare two models in ...
user avatar
0 votes
0 answers
7 views

Fairness of data-driven comparison: LR vs. mixed effects LR

In terms of comparability of LR vs. mixed effects LR, I wonder if it is a problem since number of variables is large (so far 55). However, the RF model is modelled using 55 variables, but the mixed ...
user avatar
0 votes
0 answers
20 views

R: How to determine lmer baseline in k-fold cross validation

For testing whether a predictor improves a linear mixed-effects model fitted with lme4, I used to fit a baseline model and the baseline model with an additional ...
user avatar
  • 33
0 votes
0 answers
17 views

model to explore correlation of pollution long-term exposure with genetic mutation rates

I have pollution data of several US cities, as example NY, Boston, and Chicago. I need to ...
user avatar
  • 207
0 votes
0 answers
25 views

Can we compare whether different groupings of data improve model accuracy?

I have data from 100 different lab incubations of manure samples. For each sample, a 3-day incubation was done, measuring values (y-axis) against time (x-axis). I want to perform a linear regression ...
user avatar
1 vote
0 answers
34 views

Is there a statistical test for matrix symmetry? [closed]

I have collected some data and done some processing until the question I'm faced with is - "is the data matrix $X$ a symmetric matrix?". Note that elements of $X$ represents event counts. ...
user avatar
  • 121
0 votes
0 answers
13 views

How to compare different LSTM models with different features?

I have 3 LSTM models with different features. LSTM1 is trained using Adjusted Close LSTM2 is trained using Adjusted Close and <...
user avatar
0 votes
0 answers
30 views

Comparing models to pick the best one

I am trying to compare these 5 models but am unsure of which is best. Model 1 has the worst fit and model 5 has a p value above 0.05 so i have ruled these out. Im not sure if i should go on R-squared ...
user avatar
  • 1
0 votes
0 answers
10 views

What is the number of free parameters in an n-component GMM?

I am trying to calculate BIC = -2logL + log(N)d where d is the number of free parameters or degrees of freedom. If I am fitting guassian mixture model to the data, ...
user avatar
  • 1
1 vote
1 answer
15 views

How to compare nested multilevel path models

I have two multilevel path models, one is nested in the other and I want to compare the models to see whether I should prefer the full or nested model. The models have cross-level interactions and I ...
user avatar
  • 11
1 vote
0 answers
23 views

Formal comparison of two mediation effects

Is there a method to statistically compare two mediation effects, given the same IV and DV? We have two different mediators and we want to compare the resulting models, as in: Model 1: $X – M_{1} – Y$ ...
user avatar
  • 313
1 vote
0 answers
16 views

Model comparison: what to do when reduced model doesn't converge?

This is a general question about how to do hypothesis testing via model comparison: I want to test the significance of several different predictors in a data set using model comparison. After a lot of ...
user avatar
  • 21
1 vote
0 answers
24 views

Comparing effect size for regression coefficients for models with different dependent variables

I want to compare the regression coefficients $\beta_{1}$ and $\beta_{3}$ in the two models which have the same independent variables but slightly different dependent variables: $$ \Delta{y}_{1}=\...
user avatar
1 vote
0 answers
17 views

Comparing models with partially observed data and limited supervision

Setup: Suppose we have $n$ questions and for each question $m$ answers. For each question, a model, Model A, selects one answer out of $m$. Note that for each question, zero, one, or more answers ...
user avatar
  • 670
2 votes
0 answers
21 views

Model selection: comparing Bayesian models with likelihood vs likelihood-free (Approximate Bayesian Computation)

I have two families of models that can possibly explain the data at hand. One family is rather process-based, using fairly complicated simulations and Approximate Bayesian Computation to estimate the ...
user avatar
1 vote
1 answer
39 views

GARCH model sensitivity to distribution assumptions

I am trying to fit an ARMA(4,4)- GARCH(1,1) model to return data, where the distribution of returns is highly leptokurtic. I plan to see whether autocorrelations exist in the data even after ...
user avatar
0 votes
0 answers
9 views

Comparing model fit on data aggregated at different levels

I'm fitting a non-linear model to data aggregated at 5 different levels and want to deduce at which level the model fits the data best, i.e. compare the model fits at the 5 different aggregation ...
user avatar
0 votes
0 answers
20 views

Is it okay to use AIC values generated from summary() for glmer?

I am conducting a random effect logistic regression using glmer in R. I have 13 different predictor variables, which I am evaluating at 4 different spatial scales (i.e. I have 13 variables derived ...
user avatar
0 votes
0 answers
47 views

Why does a Decision Tree algorithm outperform the Random Forest Algorithm in certain cases?

Currently I write my master thesis that deals with the binary prediction of university dropouts (dropout - yes/no). In the thesis, I compare the performance of three different classification ...
user avatar
0 votes
0 answers
21 views

Time Series Forecasting with Different Time Horizon for Comparing Models?

At the moment I am dealing with a time series problem. The data I have is about 6 years and in daily frequency. I want to try out different models on the data and I came up with an experiment: ...
user avatar
0 votes
0 answers
9 views

Combination of multiple datasets to compare models with experiments

I work on metal fatigue lifetime estimation of material under different loading conditions. I have performed a series of experiments on a certain material to determine the lifetime of the material for ...
user avatar
  • 1
0 votes
0 answers
16 views

what is the difference between RMSE and Mc error (in Bayesian estimation)?

Please, I want to know if there is a difference between Root mean square error (RMSE) and Monte carlo error (Mc error). I need to compare my models but in Bayesian modelling I found a value like Mc ...
user avatar
1 vote
0 answers
21 views

Is there a consensus on bad practice when doing model comparison in deep learning?

Lately I've been thinking about how to maintain the Equity property when Benchmarking DL for research. On Wikipedia, Equity property is defined as "All systems should be fairly compared." ...
user avatar
0 votes
0 answers
15 views

Null model comparisons with mixed models

I have a question about the general practice of null model comparisons. I understand why they can be useful in assessing model quality for prediction. My question comes when involving varying (random) ...
user avatar
  • 51
1 vote
0 answers
46 views

Goodness of fit of two models

My two models that I would like to compare are Weibull distribution and Quantile regression (in the case of Weibull distribution I have estimated the quantile curves). I would like to know if there is ...
user avatar
  • 161
1 vote
3 answers
112 views

How to compare two random forests in scikit-learn?

With most learning algorithms, one can compare the models resulting from applying the algorithm on samples of data by the parameters of the models. For example, one can compare two logistic regression ...
user avatar
  • 13
0 votes
0 answers
10 views

Creating a module to compare available forecasting techniques

I want to prepare a module for comparison of all forecasting techniques - beginning with classical approaches (arima, ses, also prophet etc.), then moving to sophisticated ML techniques (like LSTM, ES-...
user avatar
0 votes
1 answer
135 views

How I can compare three models (nested) using lavaan with R?

I want to test a new model, and for doing this, as I only have cross sectional data, I need to compare different models. My model is like this: ...
user avatar
  • 1
0 votes
0 answers
31 views

Test effects of multiple categorical variables in a nonlinear model

I am fitting forest biomass data to a modified version of the Chapman-Richards function using the nlsLM() function. I am curious about the impact of multiple binary predictor variables on the model. ...
user avatar
1 vote
0 answers
30 views

Using previous data in Bayesian change point detection

I'm interested in the detection of a single change point. I have phase I data, $X_{-n},\ldots, X_{-1}$, and I when subsequent observation arrive I want to find if their distribution has changed (...
user avatar
  • 1,035
0 votes
0 answers
17 views

Can you compare different models created with ARTool?

I would like to do model comparisons using AIC with the package ARTool art(), which only does significance testing of the fixed effects. Is this possible? I am running an aligned rank transform ANOVA (...
user avatar
  • 1
0 votes
0 answers
16 views

How can it be that two models differ significantly with repsect to R2 but not RMSE?

I have two models. Wilcoxon rank sum test says that the RMSE of these models (10-fold cross-validation) is not significant, but it is when using R2 instead of RMSE. How can this be? Could it be that ...
user avatar
  • 2,482
1 vote
0 answers
73 views

Why is my ANOVA returning negative F Values?

this is my first post here but I have benefited from reading all of your past conversations. I will try to make this question as clear as possible. I'm trying to find if there are differences between ...
user avatar
  • 11
0 votes
0 answers
59 views

Is it reasonable to compare models by confidence interval width?

I am estimating values for the number of users holding a stock. Afterwards, I subtract the number of users that actually hold the stock in the time period from it. Let's call the result of the ...
user avatar
0 votes
0 answers
24 views

Chisq test for significance of intercept in R

I have a logistic mixed-effects model with both fixed and random effects. Imagine something like: ...
user avatar
  • 11
2 votes
1 answer
42 views

(Soft question) Is one hot encoding preferable only in models where you multliply the feature by some coefficient?

Suppose you have linear model and a single feature named "color" (for the sake of simplicity). In linear model you look for a coefficient $\theta_1$ which is going to multiply this feature $...
user avatar
  • 481
0 votes
0 answers
60 views

Comparing logistic regression models from two data sets when a parameter isn't varied in one data set but is in the other

I want to determine if a logistic regression model makes good predictions for a data set not used in its fitting with a hypothesis test; I'll call it the "new" data. One could say that the ...
user avatar
  • 1,023
0 votes
0 answers
27 views

Joint or separate regressions when groups have independent variables with distinct values

I have three independent variables x1, x2, x3 which are proportions summing up to 1, so I am only using x2 and x3 as the independent variables. The dependent variable is 3 level categorial variable. I ...
user avatar
  • 43
5 votes
1 answer
343 views

Why do you need non-linear regression if you can use a linear one to fit any kind of curvature to your data?

Polynomial regression fits a non-linear model to the data. But as a statistical estimation problem it's still linear in the sense that the regression function $h\left(\Theta, X\right)$ is linear in ...
user avatar
  • 481

1
2 3 4 5
11