Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Test effects of multiple categorical variables in a nonlinear model

I am fitting forest biomass data to a modified version of the Chapman-Richards function using the nlsLM() function. I am curious about the impact of multiple binary predictor variables on the model. ...
1
vote
0answers
9 views

Using previous data in Bayesian change point detection

I'm interested in the detection of a single change point. I have phase I data, $X_{-n},\ldots, X_{-1}$, and I when subsequent observation arrive I want to find if their distribution has changed (...
0
votes
0answers
10 views

Can you compare different models created with ARTool?

I would like to do model comparisons using AIC with the package ARTool art(), which only does significance testing of the fixed effects. Is this possible? I am running an aligned rank transform ANOVA (...
0
votes
0answers
13 views

How can it be that two models differ significantly with repsect to R2 but not RMSE?

I have two models. Wilcoxon rank sum test says that the RMSE of these models (10-fold cross-validation) is not significant, but it is when using R2 instead of RMSE. How can this be? Could it be that ...
1
vote
0answers
34 views

Why is my ANOVA returning negative F Values?

this is my first post here but I have benefited from reading all of your past conversations. I will try to make this question as clear as possible. I'm trying to find if there are differences between ...
0
votes
0answers
30 views

Is it reasonable to compare models by confidence interval width?

I am estimating values for the number of users holding a stock. Afterwards, I subtract the number of users that actually hold the stock in the time period from it. Let's call the result of the ...
0
votes
0answers
18 views

Chisq test for significance of intercept in R

I have a logistic mixed-effects model with both fixed and random effects. Imagine something like: ...
2
votes
1answer
28 views

(Soft question) Is one hot encoding preferable only in models where you multliply the feature by some coefficient?

Suppose you have linear model and a single feature named "color" (for the sake of simplicity). In linear model you look for a coefficient $\theta_1$ which is going to multiply this feature $...
0
votes
0answers
17 views

Comparing logistic regression models from two data sets when a parameter isn't varied in one data set but is in the other

I want to determine if a logistic regression model makes good predictions for a data set not used in its fitting with a hypothesis test; I'll call it the "new" data. One could say that the ...
0
votes
0answers
21 views

Joint or separate regressions when groups have independent variables with distinct values

I have three independent variables x1, x2, x3 which are proportions summing up to 1, so I am only using x2 and x3 as the independent variables. The dependent variable is 3 level categorial variable. I ...
5
votes
1answer
268 views

Why do you need non-linear regression if you can use a linear one to fit any kind of curvature to your data?

Polynomial regression fits a non-linear model to the data. But as a statistical estimation problem it's still linear in the sense that the regression function $h\left(\Theta, X\right)$ is linear in ...
3
votes
2answers
107 views

How is the relationship between two variables $X$ and $Y$ supposed to "explain" $R^2\text%$ of the variation of the data?

Suppose we have a linear regression and we calculate $R^2 = 0.81$. What do we mean when we say "the relationship between two variables $X$ and $Y$ explains $81\text%$ of the variation of the data&...
1
vote
1answer
31 views

Why to put variance around the mean line to the definition of $R^2$? By what is this particular choice dictated?

Suppose we have a linear regression and we calculate $R^2 = 0.81$. That means $81\text %$ less variance around the regression line than mean line, since $R^2 = \frac{\mathrm{Var\ (mean\ line) - Var\ (...
1
vote
0answers
32 views

Is the following methodology to find the error metrics between the two curves technically sound?

I am comparing two curves, one of which is derived from physical experiment, while the latter is obtained from a simulation. The experimental curve was used to calibrate the simulation results. The ...
0
votes
0answers
17 views

Model comparison from the results from the mixed() function of the afex package

I am new to R and lmer. We have a project examining the effects of mimicking others’ voice (mimicry vs non-mimicry, a between-subjects fixed effect) , modality (reading vs listening, a within-subjects ...
0
votes
1answer
46 views

What would you do? Higher R without interaction terms

I'm working with RStudio. I've searched, but I haven't been able to find anyone with this problem. I'm dealing with a dataset that has many variables, and I've found that the model I've created with ...
13
votes
2answers
991 views

What if there is no true data-generating process?

I've been engaging in a number of forecasting efforts recently, and have rediscovered a well-known truth: That combinations of different forecasts are generally better than the forecasts themselves. ...
0
votes
1answer
43 views

Marginal likelihood for linear model with random effects to do Bayesian model comparison

Suppose I have behavioral data from multiple participants to four different conditions (four observations per participant per condition). The conditions can be characterized in terms of two fixed ...
0
votes
0answers
12 views

AIC comparison between ETS and ARIMA

I was wondering is it relevant to compare AICc on training set between ARIMA and ETS method.
0
votes
1answer
47 views

Using AIC vs Likelihood Ratio test for comparing Lognormal and Powerlaw distributions

I am interested in comparing whether a lognormal or a power law are a better fit for a given set of data. Both distributions have been fit using MLE, with $x_{min}$ determined using KS-minimization a ...
5
votes
0answers
42 views

Is it wrong to compare multiple models on the same test set and choose the best model?

Suppose we split a dataset into 3 parts (train, validation, and test). I know that it's important to make sure the test set doesn't influence our decisions during model selection or hyperparameter ...
1
vote
0answers
36 views

Help understanding Bayesian Regression versus Frequentist Regression

I'm trying to wrap my head around this and hope that someone can maybe explain it to me in a simple way. I built two models: A frequentist regression A Bayesian regression with some priors Here is ...
0
votes
0answers
12 views

Compare fit of data to different non-linear models

I have a dataset with two different groups, both are have an increasing continuous response with an increasing predictor variable. See figure below: But I want to statisticly test whether the curve ...
0
votes
0answers
29 views

How to build a saturated model for logistic regression in R?

I want to build a saturated model in R for my logistic regression. I already tried the following code obtained from another question posted here (Logistic Regression : How to obtain a saturated model):...
1
vote
1answer
35 views

Comparison of statistical models with different target variable definitions

Situation: I have a couple of statistical binary classification models (e.g. logistic regression, xgboost, random forest) and want to compare the model probabilities with each other. For simplicity, ...
0
votes
0answers
21 views

Cross-validation: matching folds when comparing two algorithms

I want to compare two algorithms using cross-validation on the same dataset. Should the data in each fold match across the two models? (i.e. training set of fold i of model 1 = training set of fold i ...
2
votes
1answer
66 views

mixed linear models comparison

When I get 2 mixed linear Models for a comparison between them For example, (B, C, D are factors) Mdoel 1 <- lmer(A ~ B * C * D + (1|individual), data = Data_1) in script Outcome in console: A ~ B *...
0
votes
1answer
31 views

Comparing test and validation ROC curves statistically

So I have two sets of data my validation set and my test set. I have a neural network model that was tuned on the validation set and then finally tested on the test set and I got similar results in ...
0
votes
0answers
29 views

model comparison of two regression models

i am trying to compare two regression models with different settings: Model 1: Y ~ X + M + C, and Model 2: M ~ X + Y + C The purpose is to check which model is better. I think likelihood based methods ...
0
votes
0answers
20 views

RMSE and MAE for comparing pearson correlation coefficients?

Lets say I have a measured set B and predicted set C, I compute their Pearson correlation coefficient with A which is the ground truth set and let these coefficients be r1 and r2. I wanted to ...
1
vote
0answers
45 views

Comparing AUPRC scores in case of different baselines

I have some imbalanced data for binary classification, which I have preprocessed in 2 different ways. That led to having a different number of observations and pos/neg ratio. Then I trained the same ...
3
votes
1answer
38 views

Methods for comparing two different models on similar problem domains

Suppose I provide predictive models estimating the likelihood someone (for the sake of argument, say recent college graduates only) defaults on a mortgage loan. One of my customers, before having had ...
0
votes
0answers
16 views

problems computing c-hat with a binomial distribution for a glm (AICcmodavg)

I am trying to run a model comparison for generalised linear models in R using the AICcmodavg package. This is one of my models: ...
0
votes
0answers
22 views

intepreting AIC and drop1 of models

Consider the following case: we have continuous response A, and indicator B,C,D. ...
0
votes
1answer
21 views

Train data selection for model comparison

Let I have data to compare two models. My first model wil be based on Arificial Neural Network(ANN) My second model will based on linear regression(LR) For the ANN, I have to divide data into 3 ...
0
votes
0answers
26 views

Classification model - metrics to optimise so that number of Predicted positives equals the number of Actual positives

I am working on a classification problem, predicting if people travel overseas or not in the future, and have been using cross-validation to tune a model. I'm trying to decide on the best metric to ...
0
votes
2answers
43 views

Rejecting a model in favor of another

Let's say currently there is a specific time series forecasting model A in use for some case and I want to test and prove whether another model B is significantly better and should replace model A. ...
1
vote
0answers
69 views

Comparing two multi-class machine learning classifiers using Stuart Maxwell Test

I need to compare 2 multi-class classifiers. So, to assess whether the difference between the two are statistically significant I have taken the following steps: obtain prediction on test data using ...
3
votes
1answer
41 views

Comparing the accuracy of binary classifiers using iterated cross-validation

Let's say that you want to compare two binary classifiers (e.g., LDA and linear SVM) for a given research question, the question being "which one will probably perform best for the problem at ...
1
vote
0answers
23 views

Bayesian comparison for two linear models with same response variable but different predictors?

I ran several separate linear models with the same response $y$, but with $n$ different predictors, i.e.: $y = \alpha + \beta x_0$, $y = \alpha + \beta x_1$, ..., $y = \alpha + \beta x_n$. All ...
0
votes
1answer
27 views

Comparability of classifier probabiliy estimates

Consider that you have 3 classification models ($model_1$, $model_2$ and $model_3$) that are designed to model whether or not customers are interested in different products of your company (binary ...
1
vote
1answer
99 views

How to compare two VAR time series models in R?

I have generated two VAR time series models in R for a dataset. My query is how can I compare those two models based on any kind of metrics like forecasting power :accuracy / f-1 score or something ...
0
votes
1answer
35 views

Comparing posterior predicted probabilities with "known" to be true probabilities

Following this description I have implemented a Bayesian logistic regression (BLR) model on some data. Lets say I have this kind of data: ...
4
votes
1answer
85 views

Is there a formal test for model comparison between polynomial regression and piecewise/segmented regression? (i.e., are these models nested?)

I am interested in comparing three models a linear regression model: mod_linear <- lm (dv ~ iv) a polynomial regression model: mod_polynomial <- lm (dv ~ iv + I(iv^2)) a one-breakpoint ...
1
vote
0answers
14 views

Which model to select of two similarly performing, models with similar architecture and number of parameters, but different depths

I am training U-Net models for two-class semantic segmentation (foreground/background). I have tested different depths of the U-Net along with different number of filters in the first conv-layer (the ...
0
votes
0answers
35 views

Comparing the marginal effect of a GLM to the OLS estimates

My question is, whether there is any way to (somewhat) compare the marginal effect of a GLM estimate to an OLS estimate. As in, "since the OLS and GLM results are very similar, I will favour OLS ...
0
votes
1answer
97 views

Comparing the marginal effects of glm output to polr output

I have a dependent variable that is technically ordinal, so I ran a ordered probit model (polr). However, an ordered probit model does not produce any residuals ...
0
votes
0answers
26 views

Validity of time series mean comparisons

I am looking to compare time series obtained from physiological responses to different treatments. I know that I should model these time series using common models, eg. AR, ARIMA, etc., and then ...
0
votes
0answers
14 views

Is it legitimate to compare likelihood ratios from different datasets?

I have two nonlinear models, $M_1$ and $M_2$, where $M_2$ has all parameters of $M_1$ and a few additional ones. Since $M_2$ is more expressive than $M_1$, it will always be at least as good as $M_1$ ...
1
vote
2answers
274 views

How to prove statistical significance, python

I'm new in statistics and would be so grateful if you give me some insights: I have two big tables - results of work Model_1 and Model_2. I created and calculated statistics - something like Precision ...

1
2 3 4 5
11