Comparing two or more models fit to a common data. Also known as "model selection".

learn more… | top users | synonyms

0
votes
1answer
8 views

Choosing smoothing parameters across multiple Naïve Bayes classifiers with different number of categories

I would like to train multiple Naïve Bayes classifiers with different number of categories, and also have a global threshold for how certain one classifier must be in order for the classification to ...
0
votes
0answers
9 views

Experimental comparison and variants [migrated]

Could anybody explain this code from Luis Torgo (the DMwR package): ...
1
vote
1answer
29 views

Consequences of violating assumptions of nonlinear regression when comparing models and/or datasets

I have a question about the consequences of using non-linear regression when the data violate the assumptions of (1) homoscedasticity and (2) normal distribution. Specifically, I am wondering about ...
1
vote
1answer
47 views

Model selection and comparison in GAMM using R (mgcv)

I'm fitting a GAMM with correlation structure, using a non-Gaussian family. Here's an example of my global model: ...
0
votes
0answers
20 views

Calculating Marginal Data Density for VAR Model

I am currently estimating Bayesian vector autoregressive (BVAR) models and I would like to do model comparison with Bayes factors. I have read about the Gelfand-Day method, the Geweke (1999) modified ...
0
votes
1answer
44 views

Comparing classification algorithms using cross validation and caret's train

I am having issues understanding some concepts of algorithm comparison/parameter optimization/cross-validation in R Let's say I want to compare two classification algorithms, such as Random Forests ...
0
votes
0answers
43 views

How to find representative distribution among distributions

I have an agent-based economic model that is simulated with many initial conditions over a fixed period of time. I am interested in looking at a specific variable(s) (say firm size) and I get a ...
1
vote
1answer
207 views

Interpretation of (scale of) AIC, AICc and BIC when comparing different models

I'm trying to fit a model to a time series, but I am pretty confused as to which is the best. I'm looking at an arima model, and ets model and an stlf model, which each performed best within their ...
1
vote
2answers
73 views

Comparison between TS models - is AIC_C or ACF/PACF analysis better to choose by?

When building a model for a time series, I have received different recommendations - One is to choose the model with the smallest AIC_C value, e.g. do an initial ACF/PACF analysis of the raw data to ...
0
votes
0answers
30 views

In the optimal model, do I need to change 'REML=FALSE' to 'REML=TRUE'?

I did the model comparison using these three models: ...
0
votes
0answers
35 views

Comparison between ARIMA and ETS models

I have a time series that I'm fitting models to, using R. I have chosen an ARIMA model based on minimising the AIC_C values. The ETS model (ets()) was chosen based on minimising the model accuracy ...
0
votes
0answers
40 views

Comparing R2 between Different Samples

Good Evening All, I'm looking to directly compare the variance explained by the same regression model (same IVs, DV) between two different samples. The way I'm conceptualizing it is like Fisher's ...
1
vote
1answer
76 views

F-test on two regression models

How to compare two simple regression models? Say I have two regression models: Model 1: $Y= c+X_1+X_2$ Model 2: $Y= c+X_3+X_4$ How to test null hypothesis that both of these models have equal ...
1
vote
1answer
85 views

Comparing Time Series Forecast Models

I'm to write a short report on Time Series forecast comparison. I'm a beginner in the field. I want to investigate how one chooses which model is better than the other based on the forecast results. ...
1
vote
0answers
28 views

SEM Model Comparisons

I am learning/applying SEM to survey data in a behavioural study. I am comparing two structural models - one is simply having all the latent explanatory variables pointing to the response latent ...
0
votes
0answers
28 views

Analyzing variance of major parameters of SVM model

I am using SVM to classify a two class problem using the set of features from a dataset of 474 samples making 237 training, 237 test samples. I have cross validated by making 100 random combination of ...
0
votes
1answer
119 views

A way to compute significance of R-squared change across models in a path model, or specifically lavaan?

I have a straightforward path model with a single endogenous variable and multiple observed predictors - in other words, a regression. (I'm doing it as a path model to be able to easily test ...
0
votes
2answers
50 views

Choosing the best multiple regression model

I have two models, one restricted, the other full. Is there a test-statistic decision criteria I could use to make a decision as to which model might be better?
2
votes
0answers
31 views

Evaluating survival models in the presence of covariate-dependent censoring

I have a censored survival analysis problem with the following characteristics: Failure times are discretized The censorship distribution depends on certain covariates I don't have a ...
0
votes
0answers
71 views

Using lmer for random effect and nested interpreting model comparison

I'm trying to do a lmer in r. I did an experiment in which I manipulated the temperature. My dataframe contains the following factors: Site: 11 level, random Species: 2 levels, fixed Subject: 10, ...
1
vote
0answers
53 views

Model comparison: segmented GLM and GLM

I have two model comparison questions: First comparison: I have a GLM model and a segmented GLM model. both models have the same dependent and independent variables. The only difference is that we ...
0
votes
0answers
68 views

Compare 2 RMSE with different Standard Deviation

How can I compare RMSE of two models with different SD. for example I have two models 1) RMSE=3.5 , SD=6.58, and 2) RMSE=17 , SD=28, how can compare these models. thanks
2
votes
1answer
52 views

Correct numerical result in Bayesian model comparison

I was wondering how to calculate the following Bayesian model comparison. Suppose you have a couple of models: $$M_{1}: x \sim Bin(n, \pi); p \sim Be(1,1)$$ $$M_{2}: x \sim Bin(n, \pi); p \sim ...
1
vote
2answers
1k views

How can I compare two regression models?

I have two OLS regression models (in Stata): (1) Y = a + b_1 X_1 + b_2 X_2 + fixed effects + e (2) Y = a + b_1 X_1 + b_2 X_2 + b_3 (X_1 * X_2) + fixed effects + e ...
0
votes
0answers
13 views

Comparing goodness-of-fit across conditions

I have repeated-measures data. Participants complete both Stage A and Stage B (random ordering). I have two complex models, Model 1 and Model 2, which are fit using optimization techniques. Model 2 is ...
0
votes
0answers
31 views

Is there a BIC or AIC formula for correcting a G-statistic?

I am using the G-test (http://en.wikipedia.org/wiki/G-test) for scoring models with different numbers of parameters in a model comparison problem. Is there a BIC or AIC formula to correct a ...
3
votes
1answer
123 views

Bootstrapping Hmisc::rcorrp.cens for paired concordance?

As Frank Harrell says here and other places, it's better to compare two predictive models (Cox proportional hazards in this case) wrt discrimination (C-index) using the paired U-statistic ...
1
vote
1answer
170 views

How to assess the incremental additive information of a new predictor in a Cox model using R?

I want to check whether the addition of the new predictor x3 improves the predictive information of a Cox model significantly or not. So far I have: ...
0
votes
0answers
29 views

analysis strategy for selecting and/or transforming correlated continuous biomarkers to predict binary endpoint

I am given a simulation task to come up with several analysis strategies and compare their relative performances. The horizon is wide open; I appreciate all recommendations of methods and references. ...
2
votes
1answer
64 views

Metrics to compare time series results with no defined truth standard

I have several (~10) models which all predict time series results for a specific problem (hourly heat loss from a building over a year). I do not have any measured data or truth standard results to ...
5
votes
2answers
99 views

Metrics for comparing estimated lists to a 'true' list

I'm wondering what the best ways to compare (possibly ranked) lists when we know what the true ranking is and also the variable that decides the ranking. Say this is the top 10 of a certain list, we ...
1
vote
0answers
41 views

Are these distributions independent?

I'm trying to compare the medians of two non-normal distributions: $A$ and $B$. $A$ is the distribution of completion durations for one type of task $B$ is the distribution of completion durations ...
0
votes
2answers
90 views

Correct Evaluation of Random Forest on fixed training/test set

I have to test the performance of Random Forest on the same dataset (text classification) with about 118.000 instances of which about 1/3 is used for training and 2/3 is used for testing. The division ...
0
votes
1answer
43 views

Bayes' factor vs. Bayes' Discriminant Rule

When we are comparing two models against some data, will we obtain the same (set of) posterior odds for the models both when we use the Bayes' factor and when we use the discriminant rule? If not, ...
0
votes
0answers
16 views

Finding comparison metric/methods

I want to compare the performance of two algorithms. Please suggest me some metrics/methods to compare between the algorithms. Below is given a short description of the output of the algorithms. ...
2
votes
1answer
86 views

Concern with using the DIC to compare Bayesian models

The deviance information criterion (DIC) is a very popular tool for Bayesian model selection, due, in part, to its support by the BUGS platforms. However, there are some remaining limitations as ...
1
vote
0answers
381 views

F test to compare regression coefficients across different groups, using R

Imagine the following type of dataset: I have a dependent variable Y, two independent variables X and Z, and a variable that can separate the dataset in two smaller datasets. I estimated 2 ...
1
vote
0answers
24 views

HLM-related inquiry - comparing the magnitude of coefficients

I've conducted two separate HLM analyses, each with two IVs predicting to the same DV in the same sample. The problem that I've encountered is that I would like to compare the magnitude of the ...
0
votes
0answers
27 views

Is there a statistical technique to perform this comparison? How could I do this on Stata?

I've been busy on a work where I have to compare two financial econometrics models on the determinants of financial leverage (panel data). These have only few control variables and the dependent ...
0
votes
0answers
70 views

Non-nested model comparisons - how to interpret output of coxtest in R

I'm trying to compare the outputs of two non-nested models in my dataset and am using the coxtest in the R package lmtest and ...
4
votes
1answer
431 views

Residual plot for nonlinear regression

I have a couple of questions regarding performance of nonlinear regression models. Are the residuals from a nonlinear regression model supposed to be randomly distributed too (as in linear ...
4
votes
3answers
167 views

If summarizing stats from multiple models is it meaningful to report a mean AIC?

I am currently summarizing results from several groups of models. Is it meaningful to report a mean AIC for each group of models? If not then how best to give a summary measure for each model ...
0
votes
0answers
14 views

comparing same negative binomial models built with subset data

I have a dataset made our of several stacked datasets (one for each state). I want to check whether a zero inflated negative binomial model with data from an individual state is different from the ...
0
votes
2answers
389 views

Comparing two regression models

I have two regression models. lm(TEE ~ weight + gender) lm(TEE ~ BMR) How do I compare these two models and check which one ...
1
vote
2answers
112 views

Comparing AIC among models with different amounts of data

I have a data set with many missing observations for certain parameters (NA values) in it. I have been performing model selection using AIC. Based on AIC scores I have reduce the model to the form ...
0
votes
0answers
16 views

Reporting change when the baseline varies

I am discussing how growth of a business changes over a time period with a colleague. The issue is, that I want to extrapolate the impact of business investment beyond maintenance. The example is a ...
2
votes
1answer
68 views

Comparing GLM models using predict

Suppose I have two models created by calling glm() on the same data but with different formulas and/or families. Now I want to compare which model is better by ...
1
vote
0answers
152 views

using t-test to compare multiple slopes?

I work with an escape response. Basically, I want to find differences between animals that should run and animals that stay still when a stimulus is applied to them. I've been trying several ...
0
votes
1answer
33 views

Paired difference test

I am predicting missing values by using two different methods (e.g., data imputation methods). Following I am interested to test if the predicted values, obtained by means of two different methods, ...
0
votes
1answer
245 views

Model comparison with AIC based on different sample size

Let's assume I have two models M1 and M2: M1: y ~ x1 + x2 + x3 M2: y ~ x1 + x2 + x3 + x4 Since variable x4 has some missing values the sample size of M2 is ...