Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

Filter by
Sorted by
Tagged with
0
votes
0answers
4 views

Comparing functional hypotheses accounting for uncertain interpretation of their predictions

I am interested in using an information-theoretic approach (likely AIC) to compare the explanatory power of several functional hypotheses. As an example, hypothesis H1 predicts significant association ...
1
vote
0answers
8 views

Comparing models when their performance may depend on a continuous variable

I am interested in using an information-theoretic approach (likely AIC) to compare the fit of several models to a dependent variable X. M1 may take the form ...
0
votes
1answer
26 views

Likelihood ratio not significant but bootsrapping is (and is not)

I modelled eight lmer models via the lme4 package. First, I compared the models via a likelihood ratio test yielding this: ...
0
votes
0answers
5 views

Bayesian evaluation if partitioning is justified for a dataset

I'd like to comparare whether partitioning of a dataset is justified. The data is categorical with two levels and the fitted parameter is the prevalence of positives for a certain condition in each ...
2
votes
1answer
21 views

Comparing different specifications of GARCH models with different distributional assumptions

For purely educational reasons I'm currently trying to fit different types of GARCH models, varying on the order parameters as well as flavor (standard, eGARCH, iGARCH, GJR-GARCH) and different ...
1
vote
1answer
18 views

How to compare GARCH model outcomes from two equal time series

I'm writing my thesis and will sketch the scenario I try to research: I have data for my GARCH model from two periods. The input is the same, as is the length (1y). I want to compare both the outcome ...
2
votes
0answers
14 views

Is there a way to analyse main effects on quantile mixed-effect regression in the spirit of the “ANOVA” procedure (now for medians)?

I would like to analyse my data using quantile regression with random effect. The problem is I have also categorical covariates, which will "split" the output into corresponding levels. I ...
1
vote
0answers
51 views

How to compare two LASSO models - is there an equivalent to AIC/BIC?

It is often stated online that competing OLS models explaining a common dependent variable y can be compared by calculating an AIC or BIC for each fit, and that the model with the lowest value should ...
0
votes
0answers
11 views

Adjusting p values when doing several comparisons between nested linear models

I have a linear model (in R) like this one (all variables are continous): mod0 <- lm_robust(DV ~ IV1 + IV2 + IV3 + IV4, data = df) (DV is stage of acquisition ...
3
votes
1answer
17 views

If there are several fixed effects of interest, and I want to test their inclusion via LRTs, in what order should I test them?

I have various potential fixed effect predictors for a linear mixed effect model. Some are control variables and some are predictors of interest (on their own and interactions). I am interested in ...
0
votes
0answers
13 views

What to do when a fixed effect isn't significant using LRT, but is using Satterthwaite's degrees of freedom method?

I have a linear mixed-effects model with a few control variables. There is one predictor of interest. When I compare a model with and without the effect of interest using a LRT the difference isn't ...
1
vote
1answer
13 views

Should test MSE be bootstrapped to compare fits?

Suppose you have a training and testing set. You fit two models, A and B, to the training set. They you predict on the testing set. You find (in this contrived example): Test MSE model A: 3 Test MSE ...
0
votes
0answers
14 views

what metrics can i use to compare my two classifiers?

all, i have two classifiers (xgboost and light gradient boosting) to predict if yes cancer or not. when i use roc_auc as my scoring method i get xgboost as 0.75 and light gradient boosting as 0.76. ...
0
votes
0answers
13 views

p value for difference in model outcomes

I've run two different linear mixed effects models on the same data and got two different estimates for the gradient of the longitudinal variable. e.g. model 1 has estimate 30 with standard error 5. ...
0
votes
1answer
29 views

How to assess the quality of a linear model with zero free parameters?

I have a simple linear model that predicts an outcome based on a few input variables (e.g. y = a*x + b), which are based on theory (psychology). None of the variables are free parameters, meaning ...
0
votes
0answers
9 views

Modeling slope effects to measure individual consistency

I am trying to fit a model with some pretty sparse data; I cannot collect more data so please refrain from that suggestion. I do not have a large sample size, so I have issues with singularities for ...
1
vote
1answer
27 views

RMSE(Root mean square error) value as a model comparison method

Most of times RMSE is used to compare models where: $RMSE=\sqrt{(\sum_i (y_{i,pred}-y_{i,obs})^2)/n}$ or namely $RMSE=(\sum_i (y_{i,pred}-y_{i,obs})^2/n)^{(1/2)}$ For some compared models, the ...
1
vote
1answer
12 views

How to test two methods for their capability of quantifiy target?

I have two quantitative methods A and B, and I ran a series of measurements against a known quantity of target. The difference from the expectation was 56 and 13. To check which method is truly ...
1
vote
1answer
48 views

Develop granularity-invariant criteria for comparison of logistic (binomial) models

I have a model with logistic (binomial) likelihood, with number of successes and failures as a response variable. I am comparing various models, which can be of different granularity. Different ...
1
vote
0answers
19 views

What is the motivation for using a second order approximation for KL divergence in this example? [closed]

Consider the KL divergence between two discrete distributions P and Q: with probabilities $p_1,...,p_k$, and $q_1,...,q_k$ $I^{KL}(P;Q)= \sum^{k}_{i=1}p_ilog\frac{p_i}{q_i}$. The notes then say ...
0
votes
0answers
9 views

Comparing groups, distinctly fitted with Weibull models, via the likelihood ratio test

I'm working with some data on the fatigue (i.e. time to failure under cyclic loads) of a certain alloy. I have four groups of data, each with the 'treatment' of a slightly different manufacturing ...
2
votes
0answers
34 views

Comparison of volatility models - using conditional standard deviations or conditional variances?

At the below content, I learned that; "an unbiased variance estimator's square root doesn't imply being an unbiased estimator of the standard deviation". Comparison of daily fitted volatility and ...
0
votes
0answers
11 views

How to statistically compare the coefficient of determination (R2) among multiple simple linear models (nonnested) with same scales (IV's)?

I made two simple linear regression models with the same scales (i.e. X variables, sample size and Y variable). The adjusted R2 was used to compare the good of fit between these models. But, how to ...
2
votes
0answers
13 views

Different cut aria effect in aligning and comparing two STL file with cloudcompare

I have one project for dental thesis which some aspect of it is not clear for me. In this project we need to compare one specific area of dental mold scan by cloudcompare software, so the area is ...
0
votes
0answers
19 views

volatility and conditional correlation O-GARCH

Doing an assignment i have to compare the volatility and conditional correlation of two type of O-GARCH: in the first one is standard O-GARCH while the second is an O-GARCH with principal component ...
0
votes
1answer
36 views

Can model comparison be performed on a single dummy variable of a 3-level categorical predictor?

Let's say I have some observations, Y, that I want to regress using a model containing two predictors, X and W. If I'm interested in whether adding X to the model improves it, according to some ...
0
votes
0answers
27 views

Is a fitting a logistic regression with binomial data better than fitting it with binary data?

It is known that the bernoulli distribution is a special case of the binomial distribution, and when we look at the difference between the null deviance and residual deviance are equal when fitting a ...
0
votes
0answers
15 views

retrieve best cutoff of ROCR and compare?

I am new to ROCR curves and I wanted to use them to evaluate some methods: I am using the same code as in ‘introduction to statistical learning’, that is: ...
0
votes
0answers
16 views

Compare regression and classification models. Which fits better?

I have 2 seperate approaches to a similar problem, one approach is regression focused and the other is classification based. I have the same predictor variable data set but with different outcomes (...
1
vote
0answers
9 views

can I compare orignal arima with sqrted arima model?

I use arima to fit my data. I use data and it's root sqrted respectively. Can I compare two model by comparing their AIC, loglikelihood etc.? Thanks.
0
votes
0answers
27 views

How can I compare regression coefficients from two separate regressions involving the same individuals?

Here is a toy example for a problem I am having. I am hypothesizing that: A higher occurrence of repetitive negative thoughts (repetitive_negative) will be ...
3
votes
0answers
57 views

AIC Debate Once Again…Comparison of Likelihoods Across Distributions

I am following up on this post Is it okay to compare fitted distributions with the AIC?. I've looked at the document AIC Myths. I still am not satisfied. Context: I'm working on a project in ...
0
votes
0answers
37 views

5x2cv combined F test with hyperparameter tuning

Does anything speak against using the 5x2cv combined F test combined with k-fold cross-validation in order to compare two learning algorithms? So, on each of the 10 training sets, we first apply k-...
0
votes
1answer
25 views

Accuracy : Arima Vs Feed-forward neural networks

I try to choose the best model between the Arima model and the Feed-forward neural networks. The script runs well and I use the ...
0
votes
0answers
21 views
0
votes
1answer
25 views

How to compare 2 multiple linear regression models for the same dependent variable?

I created 2 multiple linear regression models with different independent variables for the same dependent variable. I got 3 significant variables for my 1st model but only 1 significant variable for ...
4
votes
1answer
58 views

AIC model selection for group studies

In some areas, it is common to fit a model separately to multiple clusters in a data set, for instance fitting a cognitive model separately to data from each participant in an experiment. Model ...
1
vote
0answers
27 views

Which ANOVA method to use to compare variance explained between four lmer (linear mixed effects) models in R

Suppose I have four linear mixed effects models as shown: ...
1
vote
1answer
14 views

Comparing Neural Network to ARMA

I used a neural network tool in MATLAB to predict data, and it gave it's accuracy as MSE and an R-value. I used the econometricModeler tool in MATLAB to predict data using ARMA. It gave it's accuracy ...
0
votes
0answers
21 views

How to choose between these 2 Arima models

I have two Arima models with interventions. Both used automatic procedures to find the interventions. The forecasts diverge a lot, mostly because of the drift term in model 1. How can I ...
0
votes
0answers
22 views

Comparing highly correlated predictors in log-binomial regression

I'm trying to figure out the best way to assess which of my highly correlated independent variables best predicts my dependent variable (y), a binary variable coded ...
1
vote
1answer
114 views

How to compare Random Forest and Gaussian Process models on a regression problem?

I want to compare the model fitting properties of a Random Forest (RF) and a Gaussian Process (GP) on a regression problem. One way I thought of is by using mean-squared error values on a hold-out ...
0
votes
0answers
18 views

Model Comparison Tests with Bootstrapped Models

I've found myself in a catch-22 of wanting to conduct model comparison tests (like the Likelihood Ratio Test) but having to accommodate non-normal data. I know the LR test is invalid with Robust ...
0
votes
0answers
15 views

Comparing hypothesis with Wilkes Theorem

Wilk's theorem states that under certain conditions, likelihood ratios which compare two hypothesis will asymptotically conform to a chi squared distribution: $$-2* (LL(M) - LL(M')) \sim~ \chi^2(df)$$...
5
votes
1answer
52 views

Bayesian formulation of best subset regression

We know Ridge is equivalent to using a Gaussian prior and Lasso is equivalent to using a double exponential prior. What is the Bayesian interpretation (implied prior) for the best subset regression? ...
4
votes
0answers
31 views

AIC Comparison for MLM with Different Distributions

Thank you in advance for your time and consideration! I am a non-mathematically-inclined graduate student in communication just learning multilevel modeling. We are running different models - some ...
0
votes
0answers
11 views

Which significance test should be used to compare models based on Monte Carlo (holdout) cross-validation?

I have created two machine learning models and want to run significance tests on the results of various metrics (sensitivity, specificity, Cohen's kappa etc.) to see if there's any statistically ...
0
votes
0answers
49 views

Two Predictions from the same model when are they significant different?

I have a linear model and then I am using it to predict two future values. When can I claim that the two predicted values are different? e.g. ...
1
vote
1answer
24 views

ML: Model comparison using statistical tests when testset is fixed

To check if the performance difference between two method on a single dataset is statistically significant, one common approach in literature seems to be applying cross-validation (or repeated CV), ...
0
votes
0answers
15 views

in r, comparing two regression models with same number of independent variables for significant difference in prediction

I have two models that have the same dependent variable. Each model has 3 independent variables and differ on 1 of those variables (a continuous variable). I am not trying to select for which model ...

1
2 3 4 5
9