Comparing two or more models fit to a common data. Also known as "model selection".

learn more… | top users | synonyms

2
votes
0answers
10 views

Evaluating survival models in the presence of covariate-dependent censoring

I have a censored survival analysis problem with the following characteristics: Failure times are discretized The censorship distribution depends on certain covariates I don't have a ...
0
votes
0answers
31 views

Using lmer for random effect and nested interpreting model comparison

I'm trying to do a lmer in r. I did an experiment in which I manipulated the temperature. My dataframe contains the following factors: Site: 11 level, random Species: 2 levels, fixed Subject: 10, ...
1
vote
0answers
39 views

Model comparison: segmented GLM and GLM

I have two model comparison questions: First comparison: I have a GLM model and a segmented GLM model. both models have the same dependent and independent variables. The only difference is that we ...
0
votes
0answers
21 views

Compare 2 RMSE with different Standard Deviation

How can I compare RMSE of two models with different SD. for example I have two models 1) RMSE=3.5 , SD=6.58, and 2) RMSE=17 , SD=28, how can compare these models. thanks
2
votes
1answer
43 views

Correct numerical result in Bayesian model comparison

I was wondering how to calculate the following Bayesian model comparison. Suppose you have a couple of models: $$M_{1}: x \sim Bin(n, \pi); p \sim Be(1,1)$$ $$M_{2}: x \sim Bin(n, \pi); p \sim ...
1
vote
2answers
77 views

How can I compare two regression models?

I have two OLS regression models (in Stata): (1) Y = a + b_1 X_1 + b_2 X_2 + fixed effects + e (2) Y = a + b_1 X_1 + b_2 X_2 + b_3 (X_1 * X_2) + fixed effects + e ...
0
votes
0answers
12 views

Comparing goodness-of-fit across conditions

I have repeated-measures data. Participants complete both Stage A and Stage B (random ordering). I have two complex models, Model 1 and Model 2, which are fit using optimization techniques. Model 2 is ...
0
votes
0answers
11 views

Is there a BIC or AIC formula for correcting a G-statistic?

I am using the G-test (http://en.wikipedia.org/wiki/G-test) for scoring models with different numbers of parameters in a model comparison problem. Is there a BIC or AIC formula to correct a ...
3
votes
1answer
69 views

Bootstrapping Hmisc::rcorrp.cens for paired concordance?

As Frank Harrell says here and other places, it's better to compare two predictive models (Cox proportional hazards in this case) wrt discrimination (C-index) using the paired U-statistic ...
1
vote
1answer
114 views

How to assess the incremental additive information of a new predictor in a Cox model using R?

I want to check whether the addition of the new predictor x3 improves the predictive information of a Cox model significantly or not. So far I have: ...
0
votes
0answers
14 views

analysis strategy for selecting and/or transforming correlated continuous biomarkers to predict binary endpoint

I am given a simulation task to come up with several analysis strategies and compare their relative performances. The horizon is wide open; I appreciate all recommendations of methods and references. ...
2
votes
1answer
31 views

Metrics to compare time series results with no defined truth standard

I have several (~10) models which all predict time series results for a specific problem (hourly heat loss from a building over a year). I do not have any measured data or truth standard results to ...
5
votes
2answers
92 views

Metrics for comparing estimated lists to a 'true' list

I'm wondering what the best ways to compare (possibly ranked) lists when we know what the true ranking is and also the variable that decides the ranking. Say this is the top 10 of a certain list, we ...
1
vote
0answers
40 views

Are these distributions independent?

I'm trying to compare the medians of two non-normal distributions: $A$ and $B$. $A$ is the distribution of completion durations for one type of task $B$ is the distribution of completion durations ...
0
votes
2answers
66 views

Correct Evaluation of Random Forest on fixed training/test set

I have to test the performance of Random Forest on the same dataset (text classification) with about 118.000 instances of which about 1/3 is used for training and 2/3 is used for testing. The division ...
0
votes
1answer
42 views

Bayes' factor vs. Bayes' Discriminant Rule

When we are comparing two models against some data, will we obtain the same (set of) posterior odds for the models both when we use the Bayes' factor and when we use the discriminant rule? If not, ...
0
votes
0answers
13 views

Finding comparison metric/methods

I want to compare the performance of two algorithms. Please suggest me some metrics/methods to compare between the algorithms. Below is given a short description of the output of the algorithms. ...
2
votes
1answer
48 views

Concern with using the DIC to compare Bayesian models

The deviance information criterion (DIC) is a very popular tool for Bayesian model selection, due, in part, to its support by the BUGS platforms. However, there are some remaining limitations as ...
0
votes
0answers
181 views

F test to compare regression coefficients across different groups, using R

Imagine the following type of dataset: I have a dependent variable Y, two independent variables X and Z, and a variable that can separate the dataset in two smaller datasets. I estimated 2 ...
1
vote
0answers
17 views

HLM-related inquiry - comparing the magnitude of coefficients

I've conducted two separate HLM analyses, each with two IVs predicting to the same DV in the same sample. The problem that I've encountered is that I would like to compare the magnitude of the ...
0
votes
0answers
25 views

Is there a statistical technique to perform this comparison? How could I do this on Stata?

I've been busy on a work where I have to compare two financial econometrics models on the determinants of financial leverage (panel data). These have only few control variables and the dependent ...
0
votes
0answers
41 views

Non-nested model comparisons - how to interpret output of coxtest in R

I'm trying to compare the outputs of two non-nested models in my dataset and am using the coxtest in the R package lmtest and ...
4
votes
1answer
258 views

Residual plot for nonlinear regression

I have a couple of questions regarding performance of nonlinear regression models. Are the residuals from a nonlinear regression model supposed to be randomly distributed too (as in linear ...
3
votes
1answer
134 views

If summarizing stats from multiple models is it meaningful to report a mean AIC?

I am currently summarizing results from several groups of models. Is it meaningful to report a mean AIC for each group of models? If not then how best to give a summary measure for each model ...
0
votes
0answers
11 views

comparing same negative binomial models built with subset data

I have a dataset made our of several stacked datasets (one for each state). I want to check whether a zero inflated negative binomial model with data from an individual state is different from the ...
0
votes
2answers
154 views

Comparing two regression models

I have two regression models. lm(TEE ~ weight + gender) lm(TEE ~ BMR) How do I compare these two models and check which one ...
1
vote
0answers
63 views

Comparing AIC among models with different amounts of data

I have a data set with many missing observations for certain parameters (NA values) in it. I have been performing model selection using AIC. Based on AIC scores I have reduce the model to the form ...
0
votes
0answers
14 views

Reporting change when the baseline varies

I am discussing how growth of a business changes over a time period with a colleague. The issue is, that I want to extrapolate the impact of business investment beyond maintenance. The example is a ...
2
votes
1answer
49 views

Comparing GLM models using predict

Suppose I have two models created by calling glm() on the same data but with different formulas and/or families. Now I want to compare which model is better by ...
1
vote
0answers
99 views

using t-test to compare multiple slopes?

I work with an escape response. Basically, I want to find differences between animals that should run and animals that stay still when a stimulus is applied to them. I've been trying several ...
0
votes
1answer
32 views

Paired difference test

I am predicting missing values by using two different methods (e.g., data imputation methods). Following I am interested to test if the predicted values, obtained by means of two different methods, ...
0
votes
1answer
154 views

Model comparison with AIC based on different sample size

Let's assume I have two models M1 and M2: M1: y ~ x1 + x2 + x3 M2: y ~ x1 + x2 + x3 + x4 Since variable x4 has some missing values the sample size of M2 is ...
0
votes
0answers
26 views

Information Criteria: penalization for number of parameters or for within sample?

After I read chapter 7 of the new edition of Bayesian Data Analysis1, I have come to understand that while Information Criteria like DIC and WAIC are: A way measuring the adjust of the model and ...
2
votes
2answers
131 views

comparing models in R

Consider the following example: ...
1
vote
1answer
83 views

Testing that two models from two different data sets are independent

I have two different sets $(n_1=47$, $n_2=23)$ obtained under different conditions. I fit two different functions $(\text{Fu}_1$, $\text{Fu}_2)$ in MATLAB. $\text{Fu}_1$ was fit using the first data ...
2
votes
1answer
138 views

Logistic regression with mutually exclusive predictor variables

I am working on a logistic regression approach to predict the clinical status of patients (No disease vs Disease). I already have quite strong evidence indicating that the number of genes hit by a ...
0
votes
1answer
90 views

Comparing multiple classifiers over a single dataset

Im doing a comparison between 4 classifiers : A linear SVM, a random forest classifier, a multivariate gaussian and a neural network, for a dataset with 50 features and 10000 examples, the ...
2
votes
1answer
81 views

How to compare two different predictors

I have developed a new predictor based on neural networks for a specific problem in bioinformatics. This predictor takes as inputs several features and returns a boolean target value. Additionally i ...
6
votes
1answer
113 views

Dealing with correlating fixed effects in a linear mixed-effects analysis

My question is about the best way to estimate the effect of a predictor on a dependent variable, while accounting for several other predictors that may correlate with the predictor of interest. I'm ...
1
vote
0answers
57 views

Comparing effects of slightly different sets of independent variables on multiple dependent variables in logistic regression

I have a problem with logistic regression. I want to find out if there is any effect of social capital variables (civic participation, generalized trust...) on different dependent variables (namely ...
2
votes
1answer
183 views

Comparison of two logistic regression models (significant result with anova() but very similar AUCs)

I have compared two logistic regression models using the function anova(mod1,mod2,test="Chisq") in R. The result that I obtained is the following: ...
0
votes
0answers
73 views

RMSE higher in training compared to validation..what is generally acceptable?

I understand this is a bit of a vague question. I am struggling to get RMSEs similar in respect to training and test/validation when using cross validation in my models utilising partition. So in ...
0
votes
0answers
48 views

Comparison of two different RMSEs

I performed interpolation on two elevation datasets. Is it possible to compare the two RMSE values beyond just which one is larger and which one is smaller, to see if they are statistically different? ...
2
votes
0answers
181 views

How to compare models with different distributional assumptions for response variable in GLM?

Let's say I have measurements $Y$ which are all positive, and the distribution seems to be somewhat skewed. I'm modelling $Y$ in GLM framework. Now I could set my GLM using different distributional ...
0
votes
2answers
471 views

Compare regression coefficient for different dependent variables

I am currently reviewing a paper in which the authors perform the following analysis. They use the same set of independent variables to explain 3 different variables. The hypotheses are of the form: X ...
0
votes
1answer
201 views

How to compare two models' MSE if the answer range differs?

I have two prediction models. The first one returns answers in the range 0 to 1, where the correct answer is 0 or 1. The second returns answers in the range -1 to 1, and the correct answer is -1 or 1. ...
0
votes
0answers
38 views

Comparing very different models

I'm running an analysis and I've tried very different approaches, some frequentist and some Bayesian. Now I would like to compare the performance or fit of the models, and I was wondering how I can do ...
3
votes
0answers
91 views

Comparing linear models

My Linear Models professor said that it is possible sometimes to obtain two different linear models for the same data, such that the models cannot be compared to determine which is better. How can ...
2
votes
1answer
556 views

Question about performing k-fold CV with caret

I have read the help manual of caret carefully: see A Short Introduction to the caret Package. In its example, I found it split the data with createDataPartition before a model training. ...
0
votes
1answer
113 views

Comparing models with different number of predictors

Given that the overall F-test of a multiple regression model has an F distribution, which depends on the number of predictors in the model, I understand why you cannot compare the F-statistics from ...