Comparing two or more models fit to a common data. Also known as "model selection".

learn more… | top users | synonyms

0
votes
1answer
18 views

Statistic test for comparing two methods? [on hold]

I don't have very good knowledge about statistical testing. Accept my apology if my question sounds stupid. I have developed two methods for competing discomfort level of communities. one is static ...
0
votes
0answers
11 views

Method comparisons for trees and forests

I am looking to compare various types of models for a dataset, in order to determine which one is the most suitable. They will all be a form of decision tree, from the basic tree to random forests, ...
0
votes
0answers
14 views

Taking differences, indexes or levels in regression analysis?

I am new to econometrics and regression analysis and I am trying to make a decision on which path to follow on my regression. I have cross sectional data, and I want to estimate the impact of ...
1
vote
0answers
7 views

Compare logistic models on same customers

I have two logistic models on the same set of customers in telecom. 1. Propensity to convert from fixed line to prepaid line 2. Propensity to convert from fixed line to post-paid line. I have to ...
0
votes
0answers
11 views

Comparing a regression in two scenarios

I am looking for some answers regarding regression. I am running a regression in scenario 1 and then same construct measured for a scenario 2, the scenario is different but the constructs are the ...
0
votes
0answers
20 views

Equivalent measure to Matthews correlation coefficient, MCC, for multiclass classification

Thanks in advance for the help. MCC gives a measure of the quality of a binary classifier. I'm looking for a similar measure that can be used for a multi-class classifier. Ultimately what I would ...
1
vote
0answers
16 views

Interpretation of cross validation results when comparing models

I'm trying to solve a bio-medical image segmentation problem using a binary classifier and then a spatial smoothing (assuming continuous regions). I have: Training set of 10 3D scans, a total of ~30 ...
1
vote
1answer
18 views

Comparison of time series models

I'm trying to create a model for a series $X = \{X_1, X_2, ...\}$. I don't assume that the $X_i$ are identical distributed nor that they are independent but at least that they have something in common ...
0
votes
0answers
19 views

AIC for multiple nonlinear regression models

How do we got about using AIC for multiple nonlinear regression models ? For example: If i have a dataset with N instances, and they can be explained by a collection of 3 models where each model has ...
0
votes
1answer
33 views

What is the use of splitting dataset into training/test prior to cross-validation?

I've occasionally seen people advocate splitting the full dataset into training/test (typically a 70/30 or 80/20 split) and then running CV on the training set. I don't fully understand the point of ...
0
votes
0answers
20 views

How to interpret the R output of anova.coxph

The likelihood ratio chi^2 test is the gold standard for comparing nested Cox models. However I am not sure how to interpret the R output. ...
2
votes
1answer
26 views

How to determine similarity between histograms? (Which metric to use?!)

Hello Stack Exchange community, I'm relatively confused on what to do in this scenario. I am running an experiment that retrieves data of interest and stores it as an array. Experimental conditions ...
0
votes
1answer
17 views

Testing the utility of adding a continuous variable to a nonlinear regression.

Let’s say I have the hypothesis that soil fertility affects the relationship between weed biomass and crop biomass. One way to go about testing that hypothesis might be to model the relationship ...
0
votes
0answers
19 views

Appropriate statistic for showing advantage of a nonlinear model over its linear counterpart?

I am interested in demonstrating the utility of nonlinear model compared to its linear alternative. Since the outcome I am using is binary, the usual approach I can think of may be compare the ...
0
votes
0answers
22 views

Comparing models on two target variables

I have a dataset with two binary target variables $Y_a$ and $Y_b$. I can build two classification tree models: $\hat{Y}_a = M_a(X)$ $\hat{Y}_b = M_b(X)$ Questions: Is there a way to compare the ...
0
votes
0answers
30 views

reporting mixed effects linear regression: t statistic or model comparison?

We have a one factor three level repeated measure experiment with ratio data (reaction times). We fit a mixed effects linear model using lmer (in fact lmerTest) - maximal, with subjects and items as ...
2
votes
0answers
44 views

Approximating the marginal likelihood in Bayesian Model Comparison

Given some data $y$, my interest centers around a collection of models $\{\mathcal{M}_1,\mathcal{M}_2,\cdots,\mathcal{M}_L\}$ representing competing hypotheses about $y$. Each model $\mathcal{M}_l$ ...
1
vote
0answers
36 views

model comparison: chi2 vs. AIC (and BIC)

I've been using the approach from Appendix A from this paper. It states that if I have two fits with $\chi^2$ values $\chi^2_1(\nu_1)$ and $\chi^2_2(\nu_2)$ ($\nu$ - degree of freedom), then their ...
0
votes
0answers
10 views

Comparing logistic regression models with different predictors [duplicate]

What measure do I use to compare two logistic regression models with different predictors but the same response? y ~ x y ~ z I've used lrtest and anova before ...
0
votes
0answers
39 views

Comparing models using different sample sizes from the same longitudinal cohort

Epidemiological study: All of the 5 models I am comparing are derived by data from same longitudinal cohort. Each model contains the same IV, DV, and covariates. The difference between each model ...
2
votes
0answers
32 views

Comparing logistic regression models with paired data, population-averaged estimates and robust standard errors

I would like to ask if any of you have suggestions for comparing logistic regression models with paired data (3 observations/ID code), population-averaged estimates and robust standard errors (I am ...
0
votes
0answers
20 views

How can test corpora (with different sizes) be compared to each other via accuracy?

I need an advice regarding a text-classifiction scenario. I use a simple accuracy measure defined as #(correctly classified classes in a corpus X) / #(all docs in X). Let A, B and C be three ...
4
votes
0answers
49 views

F-test to determine whether more than two sets of data differ

Here is the context for my question: I understand that you can fit the same model to two different datasets separately and then fit the model to the datasets pooled together as a way to discern ...
0
votes
1answer
25 views

Choosing smoothing parameters across multiple Naïve Bayes classifiers with different number of categories

I would like to train multiple Naïve Bayes classifiers with different number of categories, and also have a global threshold for how certain one classifier must be in order for the classification to ...
1
vote
1answer
75 views

Consequences of violating assumptions of nonlinear regression when comparing models and/or datasets

I have a question about the consequences of using non-linear regression when the data violate the assumptions of (1) homoscedasticity and (2) normal distribution. Specifically, I am wondering about ...
1
vote
1answer
287 views

Model selection and comparison in GAMM using R (mgcv)

I'm fitting a GAMM with correlation structure, using a non-Gaussian family. Here's an example of my global model: ...
0
votes
0answers
33 views

Calculating Marginal Data Density for VAR Model

I am currently estimating Bayesian vector autoregressive (BVAR) models and I would like to do model comparison with Bayes factors. I have read about the Gelfand-Day method, the Geweke (1999) modified ...
0
votes
1answer
105 views

Comparing classification algorithms using cross validation and caret's train

I am having issues understanding some concepts of algorithm comparison/parameter optimization/cross-validation in R Let's say I want to compare two classification algorithms, such as Random Forests ...
0
votes
0answers
49 views

How to find representative distribution among distributions

I have an agent-based economic model that is simulated with many initial conditions over a fixed period of time. I am interested in looking at a specific variable(s) (say firm size) and I get a ...
1
vote
1answer
377 views

Interpretation of (scale of) AIC, AICc and BIC when comparing different models

I'm trying to fit a model to a time series, but I am pretty confused as to which is the best. I'm looking at an arima model, and ets model and an stlf model, which each performed best within their ...
1
vote
2answers
104 views

Comparison between TS models - is AIC_C or ACF/PACF analysis better to choose by?

When building a model for a time series, I have received different recommendations - One is to choose the model with the smallest AIC_C value, e.g. do an initial ACF/PACF analysis of the raw data to ...
0
votes
0answers
43 views

In the optimal model, do I need to change 'REML=FALSE' to 'REML=TRUE'?

I did the model comparison using these three models: ...
0
votes
0answers
78 views

Comparison between ARIMA and ETS models

I have a time series that I'm fitting models to, using R. I have chosen an ARIMA model based on minimising the AIC_C values. The ETS model (ets()) was chosen based on minimising the model accuracy ...
0
votes
0answers
66 views

Comparing R2 between Different Samples

Good Evening All, I'm looking to directly compare the variance explained by the same regression model (same IVs, DV) between two different samples. The way I'm conceptualizing it is like Fisher's ...
1
vote
1answer
173 views

F-test on two regression models

How to compare two simple regression models? Say I have two regression models: Model 1: $Y= c+X_1+X_2$ Model 2: $Y= c+X_3+X_4$ How to test null hypothesis that both of these models have equal ...
1
vote
1answer
177 views

Comparing Time Series Forecast Models

I'm to write a short report on Time Series forecast comparison. I'm a beginner in the field. I want to investigate how one chooses which model is better than the other based on the forecast results. ...
1
vote
0answers
33 views

SEM Model Comparisons

I am learning/applying SEM to survey data in a behavioural study. I am comparing two structural models - one is simply having all the latent explanatory variables pointing to the response latent ...
0
votes
0answers
31 views

Analyzing variance of major parameters of SVM model

I am using SVM to classify a two class problem using the set of features from a dataset of 474 samples making 237 training, 237 test samples. I have cross validated by making 100 random combination of ...
0
votes
1answer
143 views

A way to compute significance of R-squared change across models in a path model, or specifically lavaan?

I have a straightforward path model with a single endogenous variable and multiple observed predictors - in other words, a regression. (I'm doing it as a path model to be able to easily test ...
0
votes
2answers
56 views

Choosing the best multiple regression model

I have two models, one restricted, the other full. Is there a test-statistic decision criteria I could use to make a decision as to which model might be better?
2
votes
0answers
47 views

Evaluating survival models in the presence of covariate-dependent censoring

I have a censored survival analysis problem with the following characteristics: Failure times are discretized The censorship distribution depends on certain covariates I don't have a ...
0
votes
0answers
91 views

Using lmer for random effect and nested interpreting model comparison

I'm trying to do a lmer in r. I did an experiment in which I manipulated the temperature. My dataframe contains the following factors: Site: 11 level, random Species: 2 levels, fixed Subject: 10, ...
2
votes
0answers
71 views

Model comparison: segmented GLM and GLM

I have two model comparison questions: First comparison: I have a GLM model and a segmented GLM model. both models have the same dependent and independent variables. The only difference is that we ...
0
votes
0answers
94 views

Compare 2 RMSE with different Standard Deviation

How can I compare RMSE of two models with different SD. for example I have two models 1) RMSE=3.5 , SD=6.58, and 2) RMSE=17 , SD=28, how can compare these models. thanks
2
votes
1answer
53 views

Correct numerical result in Bayesian model comparison

I was wondering how to calculate the following Bayesian model comparison. Suppose you have a couple of models: $$M_{1}: x \sim Bin(n, \pi); p \sim Be(1,1)$$ $$M_{2}: x \sim Bin(n, \pi); p \sim ...
1
vote
2answers
2k views

How can I compare two regression models?

I have two OLS regression models (in Stata): (1) Y = a + b_1 X_1 + b_2 X_2 + fixed effects + e (2) Y = a + b_1 X_1 + b_2 X_2 + b_3 (X_1 * X_2) + fixed effects + e ...
0
votes
0answers
15 views

Comparing goodness-of-fit across conditions

I have repeated-measures data. Participants complete both Stage A and Stage B (random ordering). I have two complex models, Model 1 and Model 2, which are fit using optimization techniques. Model 2 is ...
0
votes
0answers
47 views

Is there a BIC or AIC formula for correcting a G-statistic?

I am using the G-test (http://en.wikipedia.org/wiki/G-test) for scoring models with different numbers of parameters in a model comparison problem. Is there a BIC or AIC formula to correct a ...
3
votes
1answer
161 views

Bootstrapping Hmisc::rcorrp.cens for paired concordance?

As Frank Harrell says here and other places, it's better to compare two predictive models (Cox proportional hazards in this case) wrt discrimination (C-index) using the paired U-statistic ...
1
vote
1answer
208 views

How to assess the incremental additive information of a new predictor in a Cox model using R?

I want to check whether the addition of the new predictor x3 improves the predictive information of a Cox model significantly or not. So far I have: ...