Comparing two or more models fit to a common data. Also known as "model selection".

learn more… | top users | synonyms

0
votes
0answers
10 views

Is there a statistical technique to perform this comparison? How could I do this on Stata?

I've been busy on a work where I have to compare two financial econometrics models on the determinants of financial leverage (panel data). These have only few control variables and the dependent ...
0
votes
0answers
16 views

Non-nested model comparisons - how to interpret output of coxtest in R

I'm trying to compare the outputs of two non-nested models in my dataset and am using the coxtest in the R package lmtest and ...
4
votes
1answer
93 views

Residual plot for nonlinear regression

I have a couple of questions regarding performance of nonlinear regression models. Are the residuals from a nonlinear regression model supposed to be randomly distributed too (as in linear ...
3
votes
1answer
105 views

If summarizing stats from multiple models is it meaningful to report a mean AIC?

I am currently summarizing results from several groups of models. Is it meaningful to report a mean AIC for each group of models? If not then how best to give a summary measure for each model ...
0
votes
0answers
10 views

comparing same negative binomial models built with subset data

I have a dataset made our of several stacked datasets (one for each state). I want to check whether a zero inflated negative binomial model with data from an individual state is different from the ...
0
votes
2answers
44 views

Comparing two regression models

I have two regression models. lm(TEE ~ weight + gender) lm(TEE ~ BMR) How do I compare these two models and check which one ...
1
vote
0answers
39 views

Comparing AIC among models with different amounts of data

I have a data set with many missing observations for certain parameters (NA values) in it. I have been performing model selection using AIC. Based on AIC scores I have reduce the model to the form ...
0
votes
0answers
13 views

Reporting change when the baseline varies

I am discussing how growth of a business changes over a time period with a colleague. The issue is, that I want to extrapolate the impact of business investment beyond maintenance. The example is a ...
2
votes
1answer
36 views

Comparing GLM models using predict

Suppose I have two models created by calling glm() on the same data but with different formulas and/or families. Now I want to compare which model is better by ...
1
vote
0answers
56 views

using t-test to compare multiple slopes?

I work with an escape response. Basically, I want to find differences between animals that should run and animals that stay still when a stimulus is applied to them. I've been trying several ...
0
votes
1answer
27 views

Paired difference test

I am predicting missing values by using two different methods (e.g., data imputation methods). Following I am interested to test if the predicted values, obtained by means of two different methods, ...
0
votes
1answer
64 views

Model comparison with AIC based on different sample size

Let's assume I have two models M1 and M2: M1: y ~ x1 + x2 + x3 M2: y ~ x1 + x2 + x3 + x4 Since variable x4 has some missing values the sample size of M2 is ...
0
votes
0answers
23 views

Information Criteria: penalization for number of parameters or for within sample?

After I read chapter 7 of the new edition of Bayesian Data Analysis1, I have come to understand that while Information Criteria like DIC and WAIC are: A way measuring the adjust of the model and ...
1
vote
2answers
105 views

comparing models in R

Consider the following example: ...
1
vote
1answer
70 views

Testing that two models from two different data sets are independent

I have two different sets $(n_1=47$, $n_2=23)$ obtained under different conditions. I fit two different functions $(\text{Fu}_1$, $\text{Fu}_2)$ in MATLAB. $\text{Fu}_1$ was fit using the first data ...
2
votes
1answer
108 views

Logistic regression with mutually exclusive predictor variables

I am working on a logistic regression approach to predict the clinical status of patients (No disease vs Disease). I already have quite strong evidence indicating that the number of genes hit by a ...
0
votes
1answer
66 views

Comparing multiple classifiers over a single dataset

Im doing a comparison between 4 classifiers : A linear SVM, a random forest classifier, a multivariate gaussian and a neural network, for a dataset with 50 features and 10000 examples, the ...
2
votes
1answer
66 views

How to compare two different predictors

I have developed a new predictor based on neural networks for a specific problem in bioinformatics. This predictor takes as inputs several features and returns a boolean target value. Additionally i ...
6
votes
1answer
96 views

Dealing with correlating fixed effects in a linear mixed-effects analysis

My question is about the best way to estimate the effect of a predictor on a dependent variable, while accounting for several other predictors that may correlate with the predictor of interest. I'm ...
1
vote
0answers
49 views

Comparing effects of slightly different sets of independent variables on multiple dependent variables in logistic regression

I have a problem with logistic regression. I want to find out if there is any effect of social capital variables (civic participation, generalized trust...) on different dependent variables (namely ...
2
votes
1answer
109 views

Comparison of two logistic regression models (significant result with anova() but very similar AUCs)

I have compared two logistic regression models using the function anova(mod1,mod2,test="Chisq") in R. The result that I obtained is the following: ...
0
votes
0answers
54 views

RMSE higher in training compared to validation..what is generally acceptable?

I understand this is a bit of a vague question. I am struggling to get RMSEs similar in respect to training and test/validation when using cross validation in my models utilising partition. So in ...
2
votes
0answers
134 views

How to compare models with different distributional assumptions for response variable in GLM?

Let's say I have measurements $Y$ which are all positive, and the distribution seems to be somewhat skewed. I'm modelling $Y$ in GLM framework. Now I could set my GLM using different distributional ...
0
votes
2answers
227 views

Compare regression coefficient for different dependent variables

I am currently reviewing a paper in which the authors perform the following analysis. They use the same set of independent variables to explain 3 different variables. The hypotheses are of the form: X ...
0
votes
1answer
112 views

How to compare two models' MSE if the answer range differs?

I have two prediction models. The first one returns answers in the range 0 to 1, where the correct answer is 0 or 1. The second returns answers in the range -1 to 1, and the correct answer is -1 or 1. ...
0
votes
0answers
38 views

Comparing very different models

I'm running an analysis and I've tried very different approaches, some frequentist and some Bayesian. Now I would like to compare the performance or fit of the models, and I was wondering how I can do ...
0
votes
0answers
44 views

Alternative denominator for F test in nested model comparison?

In comparing two nested multivariate models, the usual F test statistic is in this form: $\frac{(SSR_2 - SSR_1)/(p_2 - p_1)}{MSE_2} \sim F_{p_2-p_1, N - p_2}$ assuming model 1 is nested within model ...
3
votes
0answers
74 views

Comparing linear models

My Linear Models professor said that it is possible sometimes to obtain two different linear models for the same data, such that the models cannot be compared to determine which is better. How can ...
2
votes
1answer
277 views

Question about performing k-fold CV with caret

I have read the help manual of caret carefully: see A Short Introduction to the caret Package. In its example, I found it split the data with createDataPartition before a model training. ...
0
votes
1answer
96 views

Comparing models with different number of predictors

Given that the overall F-test of a multiple regression model has an F distribution, which depends on the number of predictors in the model, I understand why you cannot compare the F-statistics from ...
0
votes
1answer
271 views

Multiple Linear Regression Models Comparison Based on R-squared And Residual Errors

am currently working on a problem where I have to calibrate weather parameters at a ground location using Satellite data available (1979-2012) over rectangular grid points and surface observatory data ...
1
vote
0answers
59 views

R - using and F test and bootstrap to test multiple linear restrictions

I want to compare two linear models. lm3 is the full model, lm4 is the restricted model. ...
1
vote
1answer
24 views

What kind of test should I use for two different objects?

Say I'm trying to compare the speeds of 2 different cars in 5 different climates and roads. I've got the mean of each one and the standard deviation. Assuming all required assumptions are made, what ...
3
votes
1answer
787 views

How to interpret and compare models in Cox regression?

I am trying to interpret the results of a Cox regression; I am doing a PhD in medicine. I love statistics but my question is still pretty basic, I think, and I did not find an answer in previous ...
6
votes
0answers
150 views

Evaluating Time Series Prediction Performance

I have a Dynamic Naive Bayes Model trained on a couple of temporal variables. The output of the model is the prediction of P(Event) @ t+1, estimated at each ...
3
votes
1answer
141 views

Compare different statistical models that forecast the same result

Background: I'm assessing the future condition of a product where we have 18,000 units in the inventory. The product can be assessed as Poor, Fair, Good, or Excellent, based on a ...
4
votes
1answer
90 views

How can a combination of model parameters have a lower standard error than each individual coefficient?

I am investigating a potential interaction between a blood marker and a gene. I have built a cox model with two binary predictors and an interaction term for this purpose. I want to graphically ...
0
votes
0answers
30 views

Comparing models on the test data

My understanding is that if you (1) have a sufficiently large test dataset, and (2) your models have the same likelihood (noise assumption), then you should compare/select the model likelihood (or ...
2
votes
0answers
34 views

Model selection with different outcome spaces

My question is about model selection/comparison in the case of discrete outcome spaces and when the number of distinct outcomes depends on the dimension. (For the curious minded, this arises ...
2
votes
1answer
92 views

Comparing general search engine and a meta search engine. Paired t test or independent t-test?

I have built a domain-specific search engine, a meta-search engine. This search engine, of course, take its results from other search engines like Google, Bing, Yahoo and stuff. Then does some ...
0
votes
1answer
112 views

Comparing two probabilities from the same normal distribution

have a normal distribution. I would like to compare two input probabilities from this population to measure how "similar" they are. Everything is subjective but I wanted to be able to say that $x$ is ...
3
votes
2answers
173 views

How to do external validation of regression models

Very basic question here, so bear with me... I have a data set with 241 patients with 16 variables plus diagnosis (malignant vs benign). There are 3 previously published logistic regression formulas ...
3
votes
1answer
332 views

Comparison negative binomial model and quasi-Poisson

I have run negative binomial and quasi-Poisson models based on an hypothesis testing approach. My final models using both methods have different covariates and interactions. It seems that there are no ...
3
votes
0answers
346 views

Likelihood ratio test versus AIC for model comparison

When performing model comparison, why does the likelihood ratio test require two nested models while this is not required when using the AIC?
2
votes
0answers
202 views

Comparing regression coefficients across models with standardized dependent variables

The Situation: There are four *identical* spatial regression models, except each uses a different dependent variable. The independent variables consist of a standard set of variables derived from a ...
2
votes
1answer
115 views

Bayesian model comparison: What is it about MCMC that makes RSS or BIC hard to use?

I'm trying to figure out why certain methods are used for comparing models in Bayesian statistics. DIC is often used in Bayesian model comparison. However, I'm under the impression that one could ...
2
votes
0answers
215 views

Number of free parameters in Gaussian mixture models

When comparing GMM models with different number of components (i.e number of Gaussians) one penalizes the likelihood for the total number of free parameters in the mixture model. If the data is in $D$ ...
1
vote
0answers
58 views

Cross-Validation: Include IV which are significant in one model but insignificant in the other?

I ran regression analyses using SAS with two different data sets containing different individuals but exactly the same IV and DV: let's call them "low_deviance" and "high_deviance" and I would like to ...
2
votes
1answer
113 views

Is Mahalanobis distance dependent from the vector dimensions?

I would like to know if Mahalanobis distance is a normalized measure and the feature dimensions (or vector dimensions) do not affect the measure? In other words if I have a vector $ v \in \Re^n $ ...
3
votes
1answer
155 views

Evaluating parametric vs non-parametric methods

I am having a hard time finding comparisons between non-parametric and parametric methods, specifically for the task of density estimation (e.g. GMM vs using Dirichlet Processes). More than ...