Comparing two or more models fit to a common data. Also known as "model selection".

learn more… | top users | synonyms

1
vote
0answers
17 views

model comparison with negative features

I've been using the approach from Appendix A from this paper. It states that if I have two fits with $\chi^2$ values $\chi^2_1(\nu_1)$ and $\chi^2_2(\nu_2)$ ($\nu$ - degree of freedom), then their ...
0
votes
0answers
7 views

Comparing logistic regression models with different predictors [duplicate]

What measure do I use to compare two logistic regression models with different predictors but the same response? y ~ x y ~ z I've used lrtest and anova before ...
0
votes
0answers
7 views

Comparing models using different sample sizes from the same longitudinal cohort

Epidemiological study: All of the 5 models I am comparing are derived by data from same longitudinal cohort. Each model contains the same IV, DV, and covariates. The difference between each model ...
2
votes
0answers
15 views

Comparing logistic regression models with paired data, population-averaged estimates and robust standard errors

I would like to ask if any of you have suggestions for comparing logistic regression models with paired data (3 observations/ID code), population-averaged estimates and robust standard errors (I am ...
0
votes
0answers
16 views

How can test corpora (with different sizes) be compared to each other via accuracy?

I need an advice regarding a text-classifiction scenario. I use a simple accuracy measure defined as #(correctly classified classes in a corpus X) / #(all docs in X). Let A, B and C be three ...
4
votes
0answers
38 views

F-test to determine whether more than two sets of data differ

Here is the context for my question: I understand that you can fit the same model to two different datasets separately and then fit the model to the datasets pooled together as a way to discern ...
0
votes
1answer
13 views

Choosing smoothing parameters across multiple Naïve Bayes classifiers with different number of categories

I would like to train multiple Naïve Bayes classifiers with different number of categories, and also have a global threshold for how certain one classifier must be in order for the classification to ...
1
vote
1answer
43 views

Consequences of violating assumptions of nonlinear regression when comparing models and/or datasets

I have a question about the consequences of using non-linear regression when the data violate the assumptions of (1) homoscedasticity and (2) normal distribution. Specifically, I am wondering about ...
1
vote
1answer
120 views

Model selection and comparison in GAMM using R (mgcv)

I'm fitting a GAMM with correlation structure, using a non-Gaussian family. Here's an example of my global model: ...
0
votes
0answers
25 views

Calculating Marginal Data Density for VAR Model

I am currently estimating Bayesian vector autoregressive (BVAR) models and I would like to do model comparison with Bayes factors. I have read about the Gelfand-Day method, the Geweke (1999) modified ...
0
votes
1answer
66 views

Comparing classification algorithms using cross validation and caret's train

I am having issues understanding some concepts of algorithm comparison/parameter optimization/cross-validation in R Let's say I want to compare two classification algorithms, such as Random Forests ...
0
votes
0answers
44 views

How to find representative distribution among distributions

I have an agent-based economic model that is simulated with many initial conditions over a fixed period of time. I am interested in looking at a specific variable(s) (say firm size) and I get a ...
1
vote
1answer
256 views

Interpretation of (scale of) AIC, AICc and BIC when comparing different models

I'm trying to fit a model to a time series, but I am pretty confused as to which is the best. I'm looking at an arima model, and ets model and an stlf model, which each performed best within their ...
1
vote
2answers
85 views

Comparison between TS models - is AIC_C or ACF/PACF analysis better to choose by?

When building a model for a time series, I have received different recommendations - One is to choose the model with the smallest AIC_C value, e.g. do an initial ACF/PACF analysis of the raw data to ...
0
votes
0answers
31 views

In the optimal model, do I need to change 'REML=FALSE' to 'REML=TRUE'?

I did the model comparison using these three models: ...
0
votes
0answers
53 views

Comparison between ARIMA and ETS models

I have a time series that I'm fitting models to, using R. I have chosen an ARIMA model based on minimising the AIC_C values. The ETS model (ets()) was chosen based on minimising the model accuracy ...
0
votes
0answers
49 views

Comparing R2 between Different Samples

Good Evening All, I'm looking to directly compare the variance explained by the same regression model (same IVs, DV) between two different samples. The way I'm conceptualizing it is like Fisher's ...
1
vote
1answer
102 views

F-test on two regression models

How to compare two simple regression models? Say I have two regression models: Model 1: $Y= c+X_1+X_2$ Model 2: $Y= c+X_3+X_4$ How to test null hypothesis that both of these models have equal ...
1
vote
1answer
122 views

Comparing Time Series Forecast Models

I'm to write a short report on Time Series forecast comparison. I'm a beginner in the field. I want to investigate how one chooses which model is better than the other based on the forecast results. ...
1
vote
0answers
30 views

SEM Model Comparisons

I am learning/applying SEM to survey data in a behavioural study. I am comparing two structural models - one is simply having all the latent explanatory variables pointing to the response latent ...
0
votes
0answers
28 views

Analyzing variance of major parameters of SVM model

I am using SVM to classify a two class problem using the set of features from a dataset of 474 samples making 237 training, 237 test samples. I have cross validated by making 100 random combination of ...
0
votes
1answer
125 views

A way to compute significance of R-squared change across models in a path model, or specifically lavaan?

I have a straightforward path model with a single endogenous variable and multiple observed predictors - in other words, a regression. (I'm doing it as a path model to be able to easily test ...
0
votes
2answers
52 views

Choosing the best multiple regression model

I have two models, one restricted, the other full. Is there a test-statistic decision criteria I could use to make a decision as to which model might be better?
2
votes
0answers
38 views

Evaluating survival models in the presence of covariate-dependent censoring

I have a censored survival analysis problem with the following characteristics: Failure times are discretized The censorship distribution depends on certain covariates I don't have a ...
0
votes
0answers
82 views

Using lmer for random effect and nested interpreting model comparison

I'm trying to do a lmer in r. I did an experiment in which I manipulated the temperature. My dataframe contains the following factors: Site: 11 level, random Species: 2 levels, fixed Subject: 10, ...
1
vote
0answers
58 views

Model comparison: segmented GLM and GLM

I have two model comparison questions: First comparison: I have a GLM model and a segmented GLM model. both models have the same dependent and independent variables. The only difference is that we ...
0
votes
0answers
77 views

Compare 2 RMSE with different Standard Deviation

How can I compare RMSE of two models with different SD. for example I have two models 1) RMSE=3.5 , SD=6.58, and 2) RMSE=17 , SD=28, how can compare these models. thanks
2
votes
1answer
52 views

Correct numerical result in Bayesian model comparison

I was wondering how to calculate the following Bayesian model comparison. Suppose you have a couple of models: $$M_{1}: x \sim Bin(n, \pi); p \sim Be(1,1)$$ $$M_{2}: x \sim Bin(n, \pi); p \sim ...
1
vote
2answers
1k views

How can I compare two regression models?

I have two OLS regression models (in Stata): (1) Y = a + b_1 X_1 + b_2 X_2 + fixed effects + e (2) Y = a + b_1 X_1 + b_2 X_2 + b_3 (X_1 * X_2) + fixed effects + e ...
0
votes
0answers
13 views

Comparing goodness-of-fit across conditions

I have repeated-measures data. Participants complete both Stage A and Stage B (random ordering). I have two complex models, Model 1 and Model 2, which are fit using optimization techniques. Model 2 is ...
0
votes
0answers
36 views

Is there a BIC or AIC formula for correcting a G-statistic?

I am using the G-test (http://en.wikipedia.org/wiki/G-test) for scoring models with different numbers of parameters in a model comparison problem. Is there a BIC or AIC formula to correct a ...
3
votes
1answer
137 views

Bootstrapping Hmisc::rcorrp.cens for paired concordance?

As Frank Harrell says here and other places, it's better to compare two predictive models (Cox proportional hazards in this case) wrt discrimination (C-index) using the paired U-statistic ...
1
vote
1answer
180 views

How to assess the incremental additive information of a new predictor in a Cox model using R?

I want to check whether the addition of the new predictor x3 improves the predictive information of a Cox model significantly or not. So far I have: ...
0
votes
0answers
31 views

analysis strategy for selecting and/or transforming correlated continuous biomarkers to predict binary endpoint

I am given a simulation task to come up with several analysis strategies and compare their relative performances. The horizon is wide open; I appreciate all recommendations of methods and references. ...
2
votes
1answer
77 views

Metrics to compare time series results with no defined truth standard

I have several (~10) models which all predict time series results for a specific problem (hourly heat loss from a building over a year). I do not have any measured data or truth standard results to ...
5
votes
2answers
99 views

Metrics for comparing estimated lists to a 'true' list

I'm wondering what the best ways to compare (possibly ranked) lists when we know what the true ranking is and also the variable that decides the ranking. Say this is the top 10 of a certain list, we ...
1
vote
0answers
41 views

Are these distributions independent?

I'm trying to compare the medians of two non-normal distributions: $A$ and $B$. $A$ is the distribution of completion durations for one type of task $B$ is the distribution of completion durations ...
0
votes
2answers
94 views

Correct Evaluation of Random Forest on fixed training/test set

I have to test the performance of Random Forest on the same dataset (text classification) with about 118.000 instances of which about 1/3 is used for training and 2/3 is used for testing. The division ...
0
votes
1answer
43 views

Bayes' factor vs. Bayes' Discriminant Rule

When we are comparing two models against some data, will we obtain the same (set of) posterior odds for the models both when we use the Bayes' factor and when we use the discriminant rule? If not, ...
0
votes
0answers
16 views

Finding comparison metric/methods

I want to compare the performance of two algorithms. Please suggest me some metrics/methods to compare between the algorithms. Below is given a short description of the output of the algorithms. ...
2
votes
1answer
94 views

Concern with using the DIC to compare Bayesian models

The deviance information criterion (DIC) is a very popular tool for Bayesian model selection, due, in part, to its support by the BUGS platforms. However, there are some remaining limitations as ...
1
vote
0answers
421 views

F test to compare regression coefficients across different groups, using R

Imagine the following type of dataset: I have a dependent variable Y, two independent variables X and Z, and a variable that can separate the dataset in two smaller datasets. I estimated 2 ...
1
vote
0answers
25 views

HLM-related inquiry - comparing the magnitude of coefficients

I've conducted two separate HLM analyses, each with two IVs predicting to the same DV in the same sample. The problem that I've encountered is that I would like to compare the magnitude of the ...
0
votes
0answers
27 views

Is there a statistical technique to perform this comparison? How could I do this on Stata?

I've been busy on a work where I have to compare two financial econometrics models on the determinants of financial leverage (panel data). These have only few control variables and the dependent ...
0
votes
0answers
75 views

Non-nested model comparisons - how to interpret output of coxtest in R

I'm trying to compare the outputs of two non-nested models in my dataset and am using the coxtest in the R package lmtest and ...
4
votes
1answer
502 views

Residual plot for nonlinear regression

I have a couple of questions regarding performance of nonlinear regression models. Are the residuals from a nonlinear regression model supposed to be randomly distributed too (as in linear ...
4
votes
3answers
178 views

If summarizing stats from multiple models is it meaningful to report a mean AIC?

I am currently summarizing results from several groups of models. Is it meaningful to report a mean AIC for each group of models? If not then how best to give a summary measure for each model ...
0
votes
0answers
15 views

comparing same negative binomial models built with subset data

I have a dataset made our of several stacked datasets (one for each state). I want to check whether a zero inflated negative binomial model with data from an individual state is different from the ...
0
votes
2answers
436 views

Comparing two regression models

I have two regression models. lm(TEE ~ weight + gender) lm(TEE ~ BMR) How do I compare these two models and check which one ...
1
vote
2answers
125 views

Comparing AIC among models with different amounts of data

I have a data set with many missing observations for certain parameters (NA values) in it. I have been performing model selection using AIC. Based on AIC scores I have reduce the model to the form ...