# Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

456 questions
Filter by
Sorted by
Tagged with
2answers
6k views

### Compare regression coefficient for different dependent variables

I am currently reviewing a paper in which the authors perform the following analysis. They use the same set of independent variables to explain 3 different variables. The hypotheses are of the form: X ...
1answer
2k views

### How to compare two models' MSE if the answer range differs?

I have two prediction models. The first one returns answers in the range 0 to 1, where the correct answer is 0 or 1. The second returns answers in the range -1 to 1, and the correct answer is -1 or 1. ...
0answers
169 views

### Comparing linear models

My Linear Models professor said that it is possible sometimes to obtain two different linear models for the same data, such that the models cannot be compared to determine which is better. How can ...
1answer
3k views

### Question about performing k-fold CV with caret

I have read the help manual of caret carefully: see A Short Introduction to the caret Package. In its example, I found it split the data with createDataPartition before a model training. ...
1answer
48k views

### Difference between “in-sample” and “pseudo out-of-sample” forecasts

Is there an explicit difference between in-sample forecasts and pseudo out-of-sample forecasts. Both is meant in the context of evaluating and comparing forecasting models.
1answer
2k views

### Comparing models with different number of predictors

Given that the overall F-test of a multiple regression model has an F distribution, which depends on the number of predictors in the model, I understand why you cannot compare the F-statistics from ...
1answer
699 views

### Multiple Linear Regression Models Comparison Based on R-squared And Residual Errors

am currently working on a problem where I have to calibrate weather parameters at a ground location using Satellite data available (1979-2012) over rectangular grid points and surface observatory data ...
1answer
38 views

### What kind of test should I use for two different objects?

Say I'm trying to compare the speeds of 2 different cars in 5 different climates and roads. I've got the mean of each one and the standard deviation. Assuming all required assumptions are made, what ...
1answer
13k views

### How to interpret and compare models in Cox regression?

I am trying to interpret the results of a Cox regression; I am doing a PhD in medicine. I love statistics but my question is still pretty basic, I think, and I did not find an answer in previous ...
1answer
1k views

### Evaluating Time Series Prediction Performance

I have a Dynamic Naive Bayes Model trained on a couple of temporal variables. The output of the model is the prediction of P(Event) @ t+1, estimated at each ...
2answers
638 views

### Compare different statistical models that forecast the same result

Background: I'm assessing the future condition of a product where we have 18,000 units in the inventory. The product can be assessed as Poor, Fair, Good, or Excellent, based on a ...
1answer
225 views

### How can a combination of model parameters have a lower standard error than each individual coefficient?

I am investigating a potential interaction between a blood marker and a gene. I have built a cox model with two binary predictors and an interaction term for this purpose. I want to graphically ...
0answers
135 views

### Comparing models on the test data

My understanding is that if you (1) have a sufficiently large test dataset, and (2) your models have the same likelihood (noise assumption), then you should compare/select the model likelihood (or log-...
1answer
79 views

### Model selection with different outcome spaces

My question is about model selection/comparison in the case of discrete outcome spaces and when the number of distinct outcomes depends on the dimension. (For the curious minded, this arises ...
1answer
482 views

### Comparing general search engine and a meta search engine. Paired t test or independent t-test?

I have built a domain-specific search engine, a meta-search engine. This search engine, of course, take its results from other search engines like Google, Bing, Yahoo and stuff. Then does some ...
1answer
253 views

### Comparing two probabilities from the same normal distribution

have a normal distribution. I would like to compare two input probabilities from this population to measure how "similar" they are. Everything is subjective but I wanted to be able to say that $x$ is ...
2answers
924 views

### How to do external validation of regression models

Very basic question here, so bear with me... I have a data set with 241 patients with 16 variables plus diagnosis (malignant vs benign). There are 3 previously published logistic regression formulas ...
1answer
5k views

### Comparison negative binomial model and quasi-Poisson

I have run negative binomial and quasi-Poisson models based on an hypothesis testing approach. My final models using both methods have different covariates and interactions. It seems that there are no ...
0answers
2k views

### Likelihood ratio test versus AIC for model comparison

When performing model comparison, why does the likelihood ratio test require two nested models while this is not required when using the AIC?
1answer
1k views

### Comparing regression coefficients across models with standardized dependent variables

The Situation: I have four similar spatial regression models, differing only in their dependent variables. The independent variables consist of a standard set of variables derived from a principal ...
1answer
418 views

### Bayesian model comparison: What is it about MCMC that makes RSS or BIC hard to use?

I'm trying to figure out why certain methods are used for comparing models in Bayesian statistics. DIC is often used in Bayesian model comparison. However, I'm under the impression that one could ...
1answer
3k views

### Number of free parameters in Gaussian mixture models

When comparing GMM models with different number of components (i.e number of Gaussians) one penalizes the likelihood for the total number of free parameters in the mixture model. If the data is in $D$ ...
0answers
114 views

### Cross-Validation: Include IV which are significant in one model but insignificant in the other?

I ran regression analyses using SAS with two different data sets containing different individuals but exactly the same IV and DV: let's call them "low_deviance" and "high_deviance" and I would like to ...
1answer
897 views

### Is Mahalanobis distance dependent from the vector dimensions?

I would like to know if Mahalanobis distance is a normalized measure and the feature dimensions (or vector dimensions) do not affect the measure? In other words if I have a vector $v \in \Re^n$ ...
1answer
218 views

### Evaluating parametric vs non-parametric methods

I am having a hard time finding comparisons between non-parametric and parametric methods, specifically for the task of density estimation (e.g. GMM vs using Dirichlet Processes). More than ...
1answer
850 views

### comparing AIC (or BIC or whatever) between different SETS of models

Suppose I have $m$ competing models. Suppose also that I could classify these models into $s$ sets. For example, I could classify models of migration behavior conditional on climatic conditions by the ...
2answers
1k views

### Regression and simple SEM: identification and model comparison

I have two problems I am trying to solve. I am using MPLUS, but I have very general questions for which MPLUS knowledge will not be necessary. Terminology: y = DV, x = IV (1) Simple regression (N=...
2answers
490 views

### Comparison of two simulated curves for given experimental data

Physicist here with a naive question, I've got a data set (points $x_i,y(x_i),i\in \mathbb N$) and two different smooth curves $f(x),g(x)$ from my simulation. I can't quite judge with my eye which ...
3answers
17k views

### Prerequisites for AIC model comparison

What are exactly the prerequisites, that need to be fulfilled for AIC model comparison to work? I just came around this question when I did comparison like this: ...
1answer
165 views

### Comparing two Bayesian models

To compare two Bayesian models for the same data, I plotted the fitted values of the first model versus the fitted values of the second model. The plot indicates a perfect similarity. Relying on this ...
0answers
433 views

### How to compare a frequentist model with a Bayesian one

I analyzed my data by using maximum likelihood estimation and a Bayesian approach. Now, I want to see which model has a better fit. How could I do this using either plots or numbers? Please be ...
0answers
763 views

### When to use likelihood ratio test vs. incremental F-test in nested nonlinear model selection?

I have two nested nonlinear models and I want to know which provides a better fit to some data. I see descriptions of both the likelihood ratio test and of the incremental F-test (also called the ...
2answers
58k views

### Comparing two linear regression models

I would like to compare two linear regression models which represent degradation rates of a mRNA over time under two different conditions. The data for each model collected independently. Here is ...
1answer
1k views

### How to calculate absolute fit indices (RMSEA, GFI…) from relative ones (AIC, BIC…)?

I have conducted an IRT analysis with Conquest in order to compare two models (1-dimensional vs. 8-dimensional) applied to a given data set (41 items of a questionnaire, N=195). Comparing the ...
3answers
5k views

### Similarity of two discrete fourier tranforms?

In climate modelling, you're looking for a models that can adequately portray the Earth's climate. This includes showing patterns that are semi-cyclical: things like the El Nino Southern Oscillation. ...
1answer
390 views

### Group comparison: joint model vs. separate model, contradictory result

Any hint on this issue would be really welcome: The goal is to employ a two-country comparison analysis. When running two separate OLS regressions, one for each country, the coefficient of variable x ...
0answers
279 views

### Comparing model fit with heteroscedastic data

I am developing a physiological test using R that requires some parameters optimised. In comparing the new method against the existing method, the values of individual readings correlate in a linear ...
0answers
154 views

### Open source data mining package comparsion

Is there sources where I can see performance of different ML models applied to same data? Currently I want to understand performance percision/recall difference between mahout online logistic ...
2answers
766 views

### Impact of regression normality-assumption on model comparison & prediction?

This question is a continuation of the discussion here: How to test the statistical significance for categorical variable in linear regression? Following Macro's suggestion, I started a new thread. ...
0answers
225 views

### How to compare the performence between different models?

Given a data, I'd like to utilize it by two different models, such as generalized linear model (GLM) and generalized partial linear model (GPLM), and my problem is how to compare the performance of ...
1answer
657 views

### Modification of "corrected repeated k-fold cv testā when also averaging Random Forest results across multiple sampling seeds?

I would be very grateful for any ideas concerning the following problem. I would be even more grateful if someone could point me to a literature reference proposing a solution for a scenario like the ...
0answers
441 views

### Comparing two non-linear biological models: differences in y vs. differences in slope

I'm an ecology PhD student, but unfortunately not a very mathematically sophisticated one. I am interested in ways to compare two non-linear models. I was really impressed with this answer, which ...
0answers
348 views

### How do I compare 2 models?

I would like to show that a new biomarker (let's say percent tumor volume reduction at 3 weeks of treatment estimated with automatic methods) performs better than an old one (same measure, but based ...
1answer
2k views

### Comparing logistic regressions

What is an appropriate way to test whether two logistic regressions are significantly different from one another? Essentially, I have two similar logistic regressions made from two different sets of ...
1answer
289 views

2answers
6k views

### Number of parameters in Markov model

I want to use BIC for HMM model selection: BIC = -2*logLike + num_of_params * log(num_of_data) So how do I count the number of parameters in the HMM model. ...
3answers
13k views

### Model comparison between an ARIMA model and a regression model

I'm really having trouble finding out how to compare ARIMA and regression models. I understand how to evaluate ARIMA models against each other, and different types of regression models (ie: ...

1
ā¦
6 7 8
9
10