Comparing two or more models fit to a common data. Also known as "model selection".

learn more… | top users | synonyms

1
vote
1answer
14 views

Comparison of time series models

I'm trying to create a model for a series $X = \{X_1, X_2, ...\}$. I don't assume that the $X_i$ are identical distributed nor that they are independent but at least that they have something in common ...
0
votes
0answers
16 views

AIC for multiple nonlinear regression models

How do we got about using AIC for multiple nonlinear regression models ? For example: If i have a dataset with N instances, and they can be explained by a collection of 3 models where each model has ...
0
votes
1answer
27 views

What is the use of splitting dataset into training/test prior to cross-validation?

I've occasionally seen people advocate splitting the full dataset into training/test (typically a 70/30 or 80/20 split) and then running CV on the training set. I don't fully understand the point of ...
0
votes
0answers
11 views

How to interpret the R output of anova.coxph

The likelihood ratio chi^2 test is the gold standard for comparing nested Cox models. However I am not sure how to interpret the R output. ...
2
votes
1answer
21 views

How to determine similarity between histograms? (Which metric to use?!)

Hello Stack Exchange community, I'm relatively confused on what to do in this scenario. I am running an experiment that retrieves data of interest and stores it as an array. Experimental conditions ...
0
votes
1answer
17 views

Testing the utility of adding a continuous variable to a nonlinear regression.

Let’s say I have the hypothesis that soil fertility affects the relationship between weed biomass and crop biomass. One way to go about testing that hypothesis might be to model the relationship ...
0
votes
0answers
18 views

Appropriate statistic for showing advantage of a nonlinear model over its linear counterpart?

I am interested in demonstrating the utility of nonlinear model compared to its linear alternative. Since the outcome I am using is binary, the usual approach I can think of may be compare the ...
0
votes
0answers
21 views

Comparing models on two target variables

I have a dataset with two binary target variables $Y_a$ and $Y_b$. I can build two classification tree models: $\hat{Y}_a = M_a(X)$ $\hat{Y}_b = M_b(X)$ Questions: Is there a way to compare the ...
0
votes
0answers
25 views

reporting mixed effects linear regression: t statistic or model comparison?

We have a one factor three level repeated measure experiment with ratio data (reaction times). We fit a mixed effects linear model using lmer (in fact lmerTest) - maximal, with subjects and items as ...
2
votes
0answers
39 views

Approximating the marginal likelihood in Bayesian Model Comparison

Given some data $y$, my interest centers around a collection of models $\{\mathcal{M}_1,\mathcal{M}_2,\cdots,\mathcal{M}_L\}$ representing competing hypotheses about $y$. Each model $\mathcal{M}_l$ ...
1
vote
0answers
32 views

model comparison: chi2 vs. AIC (and BIC)

I've been using the approach from Appendix A from this paper. It states that if I have two fits with $\chi^2$ values $\chi^2_1(\nu_1)$ and $\chi^2_2(\nu_2)$ ($\nu$ - degree of freedom), then their ...
0
votes
0answers
8 views

Comparing logistic regression models with different predictors [duplicate]

What measure do I use to compare two logistic regression models with different predictors but the same response? y ~ x y ~ z I've used lrtest and anova before ...
0
votes
0answers
25 views

Comparing models using different sample sizes from the same longitudinal cohort

Epidemiological study: All of the 5 models I am comparing are derived by data from same longitudinal cohort. Each model contains the same IV, DV, and covariates. The difference between each model ...
2
votes
0answers
28 views

Comparing logistic regression models with paired data, population-averaged estimates and robust standard errors

I would like to ask if any of you have suggestions for comparing logistic regression models with paired data (3 observations/ID code), population-averaged estimates and robust standard errors (I am ...
0
votes
0answers
19 views

How can test corpora (with different sizes) be compared to each other via accuracy?

I need an advice regarding a text-classifiction scenario. I use a simple accuracy measure defined as #(correctly classified classes in a corpus X) / #(all docs in X). Let A, B and C be three ...
4
votes
0answers
46 views

F-test to determine whether more than two sets of data differ

Here is the context for my question: I understand that you can fit the same model to two different datasets separately and then fit the model to the datasets pooled together as a way to discern ...
0
votes
1answer
23 views

Choosing smoothing parameters across multiple Naïve Bayes classifiers with different number of categories

I would like to train multiple Naïve Bayes classifiers with different number of categories, and also have a global threshold for how certain one classifier must be in order for the classification to ...
1
vote
1answer
58 views

Consequences of violating assumptions of nonlinear regression when comparing models and/or datasets

I have a question about the consequences of using non-linear regression when the data violate the assumptions of (1) homoscedasticity and (2) normal distribution. Specifically, I am wondering about ...
1
vote
1answer
231 views

Model selection and comparison in GAMM using R (mgcv)

I'm fitting a GAMM with correlation structure, using a non-Gaussian family. Here's an example of my global model: ...
0
votes
0answers
31 views

Calculating Marginal Data Density for VAR Model

I am currently estimating Bayesian vector autoregressive (BVAR) models and I would like to do model comparison with Bayes factors. I have read about the Gelfand-Day method, the Geweke (1999) modified ...
0
votes
1answer
94 views

Comparing classification algorithms using cross validation and caret's train

I am having issues understanding some concepts of algorithm comparison/parameter optimization/cross-validation in R Let's say I want to compare two classification algorithms, such as Random Forests ...
0
votes
0answers
49 views

How to find representative distribution among distributions

I have an agent-based economic model that is simulated with many initial conditions over a fixed period of time. I am interested in looking at a specific variable(s) (say firm size) and I get a ...
1
vote
1answer
339 views

Interpretation of (scale of) AIC, AICc and BIC when comparing different models

I'm trying to fit a model to a time series, but I am pretty confused as to which is the best. I'm looking at an arima model, and ets model and an stlf model, which each performed best within their ...
1
vote
2answers
102 views

Comparison between TS models - is AIC_C or ACF/PACF analysis better to choose by?

When building a model for a time series, I have received different recommendations - One is to choose the model with the smallest AIC_C value, e.g. do an initial ACF/PACF analysis of the raw data to ...
0
votes
0answers
37 views

In the optimal model, do I need to change 'REML=FALSE' to 'REML=TRUE'?

I did the model comparison using these three models: ...
0
votes
0answers
71 views

Comparison between ARIMA and ETS models

I have a time series that I'm fitting models to, using R. I have chosen an ARIMA model based on minimising the AIC_C values. The ETS model (ets()) was chosen based on minimising the model accuracy ...
0
votes
0answers
62 views

Comparing R2 between Different Samples

Good Evening All, I'm looking to directly compare the variance explained by the same regression model (same IVs, DV) between two different samples. The way I'm conceptualizing it is like Fisher's ...
1
vote
1answer
144 views

F-test on two regression models

How to compare two simple regression models? Say I have two regression models: Model 1: $Y= c+X_1+X_2$ Model 2: $Y= c+X_3+X_4$ How to test null hypothesis that both of these models have equal ...
1
vote
1answer
158 views

Comparing Time Series Forecast Models

I'm to write a short report on Time Series forecast comparison. I'm a beginner in the field. I want to investigate how one chooses which model is better than the other based on the forecast results. ...
1
vote
0answers
32 views

SEM Model Comparisons

I am learning/applying SEM to survey data in a behavioural study. I am comparing two structural models - one is simply having all the latent explanatory variables pointing to the response latent ...
0
votes
0answers
31 views

Analyzing variance of major parameters of SVM model

I am using SVM to classify a two class problem using the set of features from a dataset of 474 samples making 237 training, 237 test samples. I have cross validated by making 100 random combination of ...
0
votes
1answer
136 views

A way to compute significance of R-squared change across models in a path model, or specifically lavaan?

I have a straightforward path model with a single endogenous variable and multiple observed predictors - in other words, a regression. (I'm doing it as a path model to be able to easily test ...
0
votes
2answers
55 views

Choosing the best multiple regression model

I have two models, one restricted, the other full. Is there a test-statistic decision criteria I could use to make a decision as to which model might be better?
2
votes
0answers
47 views

Evaluating survival models in the presence of covariate-dependent censoring

I have a censored survival analysis problem with the following characteristics: Failure times are discretized The censorship distribution depends on certain covariates I don't have a ...
0
votes
0answers
87 views

Using lmer for random effect and nested interpreting model comparison

I'm trying to do a lmer in r. I did an experiment in which I manipulated the temperature. My dataframe contains the following factors: Site: 11 level, random Species: 2 levels, fixed Subject: 10, ...
2
votes
0answers
65 views

Model comparison: segmented GLM and GLM

I have two model comparison questions: First comparison: I have a GLM model and a segmented GLM model. both models have the same dependent and independent variables. The only difference is that we ...
0
votes
0answers
85 views

Compare 2 RMSE with different Standard Deviation

How can I compare RMSE of two models with different SD. for example I have two models 1) RMSE=3.5 , SD=6.58, and 2) RMSE=17 , SD=28, how can compare these models. thanks
2
votes
1answer
53 views

Correct numerical result in Bayesian model comparison

I was wondering how to calculate the following Bayesian model comparison. Suppose you have a couple of models: $$M_{1}: x \sim Bin(n, \pi); p \sim Be(1,1)$$ $$M_{2}: x \sim Bin(n, \pi); p \sim ...
1
vote
2answers
2k views

How can I compare two regression models?

I have two OLS regression models (in Stata): (1) Y = a + b_1 X_1 + b_2 X_2 + fixed effects + e (2) Y = a + b_1 X_1 + b_2 X_2 + b_3 (X_1 * X_2) + fixed effects + e ...
0
votes
0answers
15 views

Comparing goodness-of-fit across conditions

I have repeated-measures data. Participants complete both Stage A and Stage B (random ordering). I have two complex models, Model 1 and Model 2, which are fit using optimization techniques. Model 2 is ...
0
votes
0answers
44 views

Is there a BIC or AIC formula for correcting a G-statistic?

I am using the G-test (http://en.wikipedia.org/wiki/G-test) for scoring models with different numbers of parameters in a model comparison problem. Is there a BIC or AIC formula to correct a ...
3
votes
1answer
147 views

Bootstrapping Hmisc::rcorrp.cens for paired concordance?

As Frank Harrell says here and other places, it's better to compare two predictive models (Cox proportional hazards in this case) wrt discrimination (C-index) using the paired U-statistic ...
1
vote
1answer
189 views

How to assess the incremental additive information of a new predictor in a Cox model using R?

I want to check whether the addition of the new predictor x3 improves the predictive information of a Cox model significantly or not. So far I have: ...
0
votes
0answers
35 views

analysis strategy for selecting and/or transforming correlated continuous biomarkers to predict binary endpoint

I am given a simulation task to come up with several analysis strategies and compare their relative performances. The horizon is wide open; I appreciate all recommendations of methods and references. ...
2
votes
1answer
103 views

Metrics to compare time series results with no defined truth standard

I have several (~10) models which all predict time series results for a specific problem (hourly heat loss from a building over a year). I do not have any measured data or truth standard results to ...
5
votes
2answers
102 views

Metrics for comparing estimated lists to a 'true' list

I'm wondering what the best ways to compare (possibly ranked) lists when we know what the true ranking is and also the variable that decides the ranking. Say this is the top 10 of a certain list, we ...
1
vote
0answers
42 views

Are these distributions independent?

I'm trying to compare the medians of two non-normal distributions: $A$ and $B$. $A$ is the distribution of completion durations for one type of task $B$ is the distribution of completion durations ...
0
votes
2answers
99 views

Correct Evaluation of Random Forest on fixed training/test set

I have to test the performance of Random Forest on the same dataset (text classification) with about 118.000 instances of which about 1/3 is used for training and 2/3 is used for testing. The division ...
0
votes
1answer
44 views

Bayes' factor vs. Bayes' Discriminant Rule

When we are comparing two models against some data, will we obtain the same (set of) posterior odds for the models both when we use the Bayes' factor and when we use the discriminant rule? If not, ...
0
votes
0answers
16 views

Finding comparison metric/methods

I want to compare the performance of two algorithms. Please suggest me some metrics/methods to compare between the algorithms. Below is given a short description of the output of the algorithms. ...