Comparing two or more models fit to a common data. Also known as "model selection".

learn more… | top users | synonyms

0
votes
1answer
25 views

Model comparison with AIC based on different sample size

Let's assume I have two models M1 and M2: M1: y ~ x1 + x2 + x3 M2: y ~ x1 + x2 + x3 + x4 Since variable x4 has some missing values the sample size of M2 is ...
0
votes
0answers
13 views

Information Criteria: penalization for number of parameters or for within sample?

After I read chapter 7 of the new edition of Bayesian Data Analysis1, I have come to understand that while Information Criteria like DIC and WAIC are: A way measuring the adjust of the model and ...
1
vote
2answers
90 views

comparing models in R

Consider the following example: ...
1
vote
1answer
62 views

Testing that two models from two different data sets are independent

I have two different sets $(n_1=47$, $n_2=23)$ obtained under different conditions. I fit two different functions $(\text{Fu}_1$, $\text{Fu}_2)$ in MATLAB. $\text{Fu}_1$ was fit using the first data ...
2
votes
1answer
88 views

Logistic regression with mutually exclusive predictor variables

I am working on a logistic regression approach to predict the clinical status of patients (No disease vs Disease). I already have quite strong evidence indicating that the number of genes hit by a ...
0
votes
1answer
58 views

Comparing multiple classifiers over a single dataset

Im doing a comparison between 4 classifiers : A linear SVM, a random forest classifier, a multivariate gaussian and a neural network, for a dataset with 50 features and 10000 examples, the ...
2
votes
1answer
46 views

How to compare two different predictors

I have developed a new predictor based on neural networks for a specific problem in bioinformatics. This predictor takes as inputs several features and returns a boolean target value. Additionally i ...
6
votes
1answer
85 views

Dealing with correlating fixed effects in a linear mixed-effects analysis

My question is about the best way to estimate the effect of a predictor on a dependent variable, while accounting for several other predictors that may correlate with the predictor of interest. I'm ...
1
vote
0answers
35 views

Comparing effects of slightly different sets of independent variables on multiple dependent variables in logistic regression

I have a problem with logistic regression. I want to find out if there is any effect of social capital variables (civic participation, generalized trust...) on different dependent variables (namely ...
2
votes
1answer
75 views

Comparison of two logistic regression models (significant result with anova() but very similar AUCs)

I have compared two logistic regression models using the function anova(mod1,mod2,test="Chisq") in R. The result that I obtained is the following: ...
0
votes
0answers
34 views

RMSE higher in training compared to validation..what is generally acceptable?

I understand this is a bit of a vague question. I am struggling to get RMSEs similar in respect to training and test/validation when using cross validation in my models utilising partition. So in ...
2
votes
0answers
92 views

How to compare models with different distributional assumptions for response variable in GLM?

Let's say I have measurements $Y$ which are all positive, and the distribution seems to be somewhat skewed. I'm modelling $Y$ in GLM framework. Now I could set my GLM using different distributional ...
0
votes
2answers
95 views

Compare regression coefficient for different dependent variables

I am currently reviewing a paper in which the authors perform the following analysis. They use the same set of independent variables to explain 3 different variables. The hypotheses are of the form: X ...
0
votes
1answer
64 views

How to compare two models' MSE if the answer range differs?

I have two prediction models. The first one returns answers in the range 0 to 1, where the correct answer is 0 or 1. The second returns answers in the range -1 to 1, and the correct answer is -1 or 1. ...
0
votes
0answers
37 views

Comparing very different models

I'm running an analysis and I've tried very different approaches, some frequentist and some Bayesian. Now I would like to compare the performance or fit of the models, and I was wondering how I can do ...
0
votes
0answers
31 views

Alternative denominator for F test in nested model comparison?

In comparing two nested multivariate models, the usual F test statistic is in this form: $\frac{(SSR_2 - SSR_1)/(p_2 - p_1)}{MSE_2} \sim F_{p_2-p_1, N - p_2}$ assuming model 1 is nested within model ...
3
votes
0answers
68 views

Comparing linear models

My Linear Models professor said that it is possible sometimes to obtain two different linear models for the same data, such that the models cannot be compared to determine which is better. How can ...
1
vote
1answer
134 views

Question about performing k-fold CV with caret

I have read the help manual of caret carefully: see A Short Introduction to the caret Package. In its example, I found it split the data with createDataPartition before a model training. ...
0
votes
1answer
81 views

Comparing models with different number of predictors

Given that the overall F-test of a multiple regression model has an F distribution, which depends on the number of predictors in the model, I understand why you cannot compare the F-statistics from ...
0
votes
1answer
182 views

Multiple Linear Regression Models Comparison Based on R-squared And Residual Errors

am currently working on a problem where I have to calibrate weather parameters at a ground location using Satellite data available (1979-2012) over rectangular grid points and surface observatory data ...
1
vote
0answers
49 views

R - using and F test and bootstrap to test multiple linear restrictions

I want to compare two linear models. lm3 is the full model, lm4 is the restricted model. ...
1
vote
1answer
24 views

What kind of test should I use for two different objects?

Say I'm trying to compare the speeds of 2 different cars in 5 different climates and roads. I've got the mean of each one and the standard deviation. Assuming all required assumptions are made, what ...
3
votes
1answer
465 views

How to interpret and compare models in Cox regression?

I am trying to interpret the results of a Cox regression; I am doing a PhD in medicine. I love statistics but my question is still pretty basic, I think, and I did not find an answer in previous ...
6
votes
0answers
127 views

Evaluating Time Series Prediction Performance

I have a Dynamic Naive Bayes Model trained on a couple of temporal variables. The output of the model is the prediction of P(Event) @ t+1, estimated at each ...
3
votes
1answer
117 views

Compare different statistical models that forecast the same result

Background: I'm assessing the future condition of a product where we have 18,000 units in the inventory. The product can be assessed as Poor, Fair, Good, or Excellent, based on a ...
4
votes
1answer
83 views

How can a combination of model parameters have a lower standard error than each individual coefficient?

I am investigating a potential interaction between a blood marker and a gene. I have built a cox model with two binary predictors and an interaction term for this purpose. I want to graphically ...
0
votes
0answers
27 views

Comparing models on the test data

My understanding is that if you (1) have a sufficiently large test dataset, and (2) your models have the same likelihood (noise assumption), then you should compare/select the model likelihood (or ...
2
votes
0answers
33 views

Model selection with different outcome spaces

My question is about model selection/comparison in the case of discrete outcome spaces and when the number of distinct outcomes depends on the dimension. (For the curious minded, this arises ...
2
votes
1answer
75 views

Comparing general search engine and a meta search engine. Paired t test or independent t-test?

I have built a domain-specific search engine, a meta-search engine. This search engine, of course, take its results from other search engines like Google, Bing, Yahoo and stuff. Then does some ...
0
votes
1answer
106 views

Comparing two probabilities from the same normal distribution

have a normal distribution. I would like to compare two input probabilities from this population to measure how "similar" they are. Everything is subjective but I wanted to be able to say that $x$ is ...
3
votes
2answers
156 views

How to do external validation of regression models

Very basic question here, so bear with me... I have a data set with 241 patients with 16 variables plus diagnosis (malignant vs benign). There are 3 previously published logistic regression formulas ...
3
votes
1answer
204 views

Comparison negative binomial model and quasi-Poisson

I have run negative binomial and quasi-Poisson models based on an hypothesis testing approach. My final models using both methods have different covariates and interactions. It seems that there are no ...
3
votes
0answers
278 views

Likelihood ratio test versus AIC for model comparison

When performing model comparison, why does the likelihood ratio test require two nested models while this is not required when using the AIC?
1
vote
0answers
172 views

Comparing regression coefficients across models with standardized dependent variables

The Situation: There are four *identical* spatial regression models, except each uses a different dependent variable. The independent variables consist of a standard set of variables derived from a ...
2
votes
1answer
85 views

Bayesian model comparison: What is it about MCMC that makes RSS or BIC hard to use?

I'm trying to figure out why certain methods are used for comparing models in Bayesian statistics. DIC is often used in Bayesian model comparison. However, I'm under the impression that one could ...
2
votes
0answers
151 views

Number of free parameters in Gaussian mixture models

When comparing GMM models with different number of components (i.e number of Gaussians) one penalizes the likelihood for the total number of free parameters in the mixture model. If the data is in $D$ ...
1
vote
0answers
54 views

Cross-Validation: Include IV which are significant in one model but insignificant in the other?

I ran regression analyses using SAS with two different data sets containing different individuals but exactly the same IV and DV: let's call them "low_deviance" and "high_deviance" and I would like to ...
2
votes
1answer
105 views

Is Mahalanobis distance dependent from the vector dimensions?

I would like to know if Mahalanobis distance is a normalized measure and the feature dimensions (or vector dimensions) do not affect the measure? In other words if I have a vector $ v \in \Re^n $ ...
3
votes
1answer
150 views

Evaluating parametric vs non-parametric methods

I am having a hard time finding comparisons between non-parametric and parametric methods, specifically for the task of density estimation (e.g. GMM vs using Dirichlet Processes). More than ...
0
votes
0answers
42 views

Comparing two different regression models - within one sample, with different control variables and DVs. Can this be done?

I would like to (if possible), compare the effectiveness of some IVs at predicting two different DVs, based in two seperate HMR models. The IVs are the same in both models ('demands'), although the ...
1
vote
1answer
132 views

comparing AIC (or BIC or whatever) between different SETS of models

Suppose I have $m$ competing models. Suppose also that I could classify these models into $s$ sets. For example, I could classify models of migration behavior conditional on climatic conditions by the ...
0
votes
0answers
59 views

How to statistically compare two fibonacci algorithms?

I have two Fibonacci recursive algorithms and I want to check the efficiency comparing them with each other. Does anyone know how to do it statistically? Here's my data. Recursive_1: Recursive_2 ...
0
votes
2answers
340 views

Regression and simple SEM: identification and model comparison

I have two problems I am trying to solve. I am using MPLUS, but I have very general questions for which MPLUS knowledge will not be necessary. Terminology: y = DV, x = IV (1) Simple regression ...
3
votes
2answers
99 views

Comparison of two simulated curves for given experimental data

Physicist here with a naive question, I've got a data set (points $x_i,y(x_i),i\in \mathbb N$) and two different smooth curves $f(x),g(x)$ from my simulation. I can't quite judge with my eye which ...
1
vote
0answers
109 views

How to compare Arellano-Bond models?

I want to compare different Arellano-Bond models, but I don't know exactly how to do it and which statistics to consider. For instance, should I compare $\chi^2$ of Sargan tests? What other options ...
6
votes
1answer
824 views

Prerequisites for AIC model comparison

What are exactly the prerequisites, that need to be fulfilled for AIC model comparison to work? I just came around this question when I did comparison like this: ...
2
votes
1answer
63 views

Comparing two Bayesian models

To compare two Bayesian models for the same data, I plotted the fitted values of the first model versus the fitted values of the second model. The plot indicates a perfect similarity. Relying on this ...
0
votes
0answers
149 views

How to compare a frequentist model with a Bayesian one

I analyzed my data by using maximum likelihood estimation and a Bayesian approach. Now, I want to see which model has a better fit. How could I do this using either plots or numbers? Please be ...
3
votes
0answers
210 views

When to use likelihood ratio test vs. incremental F-test in nested nonlinear model selection?

I have two nested nonlinear models and I want to know which provides a better fit to some data. I see descriptions of both the likelihood ratio test and of the incremental F-test (also called the ...
1
vote
1answer
400 views

How to calculate absolute fit indices (RMSEA, GFI…) from relative ones (AIC, BIC…)?

I have conducted an IRT analysis with Conquest in order to compare two models (1-dimensional vs. 8-dimensional) applied to a given data set (41 items of a questionnaire, N=195). Comparing the ...