Questions tagged [model-comparison]

Comparing two or more models fit to a common data set. It can be part of the process of "model selection".

Filter by
Sorted by
Tagged with
0
votes
2answers
6k views

Compare regression coefficient for different dependent variables

I am currently reviewing a paper in which the authors perform the following analysis. They use the same set of independent variables to explain 3 different variables. The hypotheses are of the form: X ...
0
votes
1answer
2k views

How to compare two models' MSE if the answer range differs?

I have two prediction models. The first one returns answers in the range 0 to 1, where the correct answer is 0 or 1. The second returns answers in the range -1 to 1, and the correct answer is -1 or 1. ...
3
votes
0answers
169 views

Comparing linear models

My Linear Models professor said that it is possible sometimes to obtain two different linear models for the same data, such that the models cannot be compared to determine which is better. How can ...
3
votes
1answer
3k views

Question about performing k-fold CV with caret

I have read the help manual of caret carefully: see A Short Introduction to the caret Package. In its example, I found it split the data with createDataPartition before a model training. ...
12
votes
1answer
48k views

Difference between “in-sample” and “pseudo out-of-sample” forecasts

Is there an explicit difference between in-sample forecasts and pseudo out-of-sample forecasts. Both is meant in the context of evaluating and comparing forecasting models.
0
votes
1answer
2k views

Comparing models with different number of predictors

Given that the overall F-test of a multiple regression model has an F distribution, which depends on the number of predictors in the model, I understand why you cannot compare the F-statistics from ...
0
votes
1answer
699 views

Multiple Linear Regression Models Comparison Based on R-squared And Residual Errors

am currently working on a problem where I have to calibrate weather parameters at a ground location using Satellite data available (1979-2012) over rectangular grid points and surface observatory data ...
1
vote
1answer
38 views

What kind of test should I use for two different objects?

Say I'm trying to compare the speeds of 2 different cars in 5 different climates and roads. I've got the mean of each one and the standard deviation. Assuming all required assumptions are made, what ...
5
votes
1answer
13k views

How to interpret and compare models in Cox regression?

I am trying to interpret the results of a Cox regression; I am doing a PhD in medicine. I love statistics but my question is still pretty basic, I think, and I did not find an answer in previous ...
9
votes
1answer
1k views

Evaluating Time Series Prediction Performance

I have a Dynamic Naive Bayes Model trained on a couple of temporal variables. The output of the model is the prediction of P(Event) @ t+1, estimated at each ...
4
votes
2answers
638 views

Compare different statistical models that forecast the same result

Background: I'm assessing the future condition of a product where we have 18,000 units in the inventory. The product can be assessed as Poor, Fair, Good, or Excellent, based on a ...
5
votes
1answer
225 views

How can a combination of model parameters have a lower standard error than each individual coefficient?

I am investigating a potential interaction between a blood marker and a gene. I have built a cox model with two binary predictors and an interaction term for this purpose. I want to graphically ...
1
vote
0answers
135 views

Comparing models on the test data

My understanding is that if you (1) have a sufficiently large test dataset, and (2) your models have the same likelihood (noise assumption), then you should compare/select the model likelihood (or log-...
2
votes
1answer
79 views

Model selection with different outcome spaces

My question is about model selection/comparison in the case of discrete outcome spaces and when the number of distinct outcomes depends on the dimension. (For the curious minded, this arises ...
2
votes
1answer
482 views

Comparing general search engine and a meta search engine. Paired t test or independent t-test?

I have built a domain-specific search engine, a meta-search engine. This search engine, of course, take its results from other search engines like Google, Bing, Yahoo and stuff. Then does some ...
0
votes
1answer
253 views

Comparing two probabilities from the same normal distribution

have a normal distribution. I would like to compare two input probabilities from this population to measure how "similar" they are. Everything is subjective but I wanted to be able to say that $x$ is ...
6
votes
2answers
924 views

How to do external validation of regression models

Very basic question here, so bear with me... I have a data set with 241 patients with 16 variables plus diagnosis (malignant vs benign). There are 3 previously published logistic regression formulas ...
10
votes
1answer
5k views

Comparison negative binomial model and quasi-Poisson

I have run negative binomial and quasi-Poisson models based on an hypothesis testing approach. My final models using both methods have different covariates and interactions. It seems that there are no ...
3
votes
0answers
2k views

Likelihood ratio test versus AIC for model comparison

When performing model comparison, why does the likelihood ratio test require two nested models while this is not required when using the AIC?
6
votes
1answer
1k views

Comparing regression coefficients across models with standardized dependent variables

The Situation: I have four similar spatial regression models, differing only in their dependent variables. The independent variables consist of a standard set of variables derived from a principal ...
4
votes
1answer
418 views

Bayesian model comparison: What is it about MCMC that makes RSS or BIC hard to use?

I'm trying to figure out why certain methods are used for comparing models in Bayesian statistics. DIC is often used in Bayesian model comparison. However, I'm under the impression that one could ...
2
votes
1answer
3k views

Number of free parameters in Gaussian mixture models

When comparing GMM models with different number of components (i.e number of Gaussians) one penalizes the likelihood for the total number of free parameters in the mixture model. If the data is in $D$ ...
1
vote
0answers
114 views

Cross-Validation: Include IV which are significant in one model but insignificant in the other?

I ran regression analyses using SAS with two different data sets containing different individuals but exactly the same IV and DV: let's call them "low_deviance" and "high_deviance" and I would like to ...
4
votes
1answer
897 views

Is Mahalanobis distance dependent from the vector dimensions?

I would like to know if Mahalanobis distance is a normalized measure and the feature dimensions (or vector dimensions) do not affect the measure? In other words if I have a vector $ v \in \Re^n $ ...
4
votes
1answer
218 views

Evaluating parametric vs non-parametric methods

I am having a hard time finding comparisons between non-parametric and parametric methods, specifically for the task of density estimation (e.g. GMM vs using Dirichlet Processes). More than ...
2
votes
1answer
850 views

comparing AIC (or BIC or whatever) between different SETS of models

Suppose I have $m$ competing models. Suppose also that I could classify these models into $s$ sets. For example, I could classify models of migration behavior conditional on climatic conditions by the ...
0
votes
2answers
1k views

Regression and simple SEM: identification and model comparison

I have two problems I am trying to solve. I am using MPLUS, but I have very general questions for which MPLUS knowledge will not be necessary. Terminology: y = DV, x = IV (1) Simple regression (N=...
3
votes
2answers
490 views

Comparison of two simulated curves for given experimental data

Physicist here with a naive question, I've got a data set (points $x_i,y(x_i),i\in \mathbb N$) and two different smooth curves $f(x),g(x)$ from my simulation. I can't quite judge with my eye which ...
27
votes
3answers
17k views

Prerequisites for AIC model comparison

What are exactly the prerequisites, that need to be fulfilled for AIC model comparison to work? I just came around this question when I did comparison like this: ...
3
votes
1answer
165 views

Comparing two Bayesian models

To compare two Bayesian models for the same data, I plotted the fitted values of the first model versus the fitted values of the second model. The plot indicates a perfect similarity. Relying on this ...
0
votes
0answers
433 views

How to compare a frequentist model with a Bayesian one

I analyzed my data by using maximum likelihood estimation and a Bayesian approach. Now, I want to see which model has a better fit. How could I do this using either plots or numbers? Please be ...
5
votes
0answers
763 views

When to use likelihood ratio test vs. incremental F-test in nested nonlinear model selection?

I have two nested nonlinear models and I want to know which provides a better fit to some data. I see descriptions of both the likelihood ratio test and of the incremental F-test (also called the ...
12
votes
2answers
58k views

Comparing two linear regression models

I would like to compare two linear regression models which represent degradation rates of a mRNA over time under two different conditions. The data for each model collected independently. Here is ...
1
vote
1answer
1k views

How to calculate absolute fit indices (RMSEA, GFI…) from relative ones (AIC, BIC…)?

I have conducted an IRT analysis with Conquest in order to compare two models (1-dimensional vs. 8-dimensional) applied to a given data set (41 items of a questionnaire, N=195). Comparing the ...
14
votes
3answers
5k views

Similarity of two discrete fourier tranforms?

In climate modelling, you're looking for a models that can adequately portray the Earth's climate. This includes showing patterns that are semi-cyclical: things like the El Nino Southern Oscillation. ...
0
votes
1answer
390 views

Group comparison: joint model vs. separate model, contradictory result

Any hint on this issue would be really welcome: The goal is to employ a two-country comparison analysis. When running two separate OLS regressions, one for each country, the coefficient of variable x ...
3
votes
0answers
279 views

Comparing model fit with heteroscedastic data

I am developing a physiological test using R that requires some parameters optimised. In comparing the new method against the existing method, the values of individual readings correlate in a linear ...
2
votes
0answers
154 views

Open source data mining package comparsion

Is there sources where I can see performance of different ML models applied to same data? Currently I want to understand performance percision/recall difference between mahout online logistic ...
1
vote
2answers
766 views

Impact of regression normality-assumption on model comparison & prediction?

This question is a continuation of the discussion here: How to test the statistical significance for categorical variable in linear regression? Following Macro's suggestion, I started a new thread. ...
1
vote
0answers
225 views

How to compare the performence between different models?

Given a data, I'd like to utilize it by two different models, such as generalized linear model (GLM) and generalized partial linear model (GPLM), and my problem is how to compare the performance of ...
1
vote
1answer
657 views

Modification of "corrected repeated k-fold cv test” when also averaging Random Forest results across multiple sampling seeds?

I would be very grateful for any ideas concerning the following problem. I would be even more grateful if someone could point me to a literature reference proposing a solution for a scenario like the ...
4
votes
0answers
441 views

Comparing two non-linear biological models: differences in y vs. differences in slope

I'm an ecology PhD student, but unfortunately not a very mathematically sophisticated one. I am interested in ways to compare two non-linear models. I was really impressed with this answer, which ...
3
votes
0answers
348 views

How do I compare 2 models?

I would like to show that a new biomarker (let's say percent tumor volume reduction at 3 weeks of treatment estimated with automatic methods) performs better than an old one (same measure, but based ...
3
votes
1answer
2k views

Comparing logistic regressions

What is an appropriate way to test whether two logistic regressions are significantly different from one another? Essentially, I have two similar logistic regressions made from two different sets of ...
3
votes
1answer
289 views

Logical extensions after logistic regression

I have a number of predictors to use for a binary (Classes 0 and 1) classification task. Let us call them $x_1, x_2, x_3, ... x_n$. The way these are calculated, my naive heuristic assumption is that $...
8
votes
1answer
191 views

How can I compare my model to a technically invalid model?

I've created nice little nonlinear model relating survival probability to length in salmon. I fit it assuming binomial errors and minimizing the negative log likelihood. I've been asked to compare it ...
5
votes
3answers
5k views

Can you compare p-values of Kolmogorov Smirnov tests of normality of two variables to say which is more normal?

I have applied the one sample Kolmogorov Smirnov test of normality to two variables and one has a larger p value but both are greater than .05. e.g., $x_1$ (p-value) = 0.09 $x_2$ (p-value) = 0.06 ...
5
votes
0answers
2k views

Comparison of positive predictive value between two models [duplicate]

Does anybody know how to compare two positive predictive value (PPV) from two different predictive models? I found some papers reporting $\chi^2$-test or Fisher exact test but I can´t figure out how ...
12
votes
2answers
6k views

Number of parameters in Markov model

I want to use BIC for HMM model selection: BIC = -2*logLike + num_of_params * log(num_of_data) So how do I count the number of parameters in the HMM model. ...
8
votes
3answers
13k views

Model comparison between an ARIMA model and a regression model

I'm really having trouble finding out how to compare ARIMA and regression models. I understand how to evaluate ARIMA models against each other, and different types of regression models (ie: ...

1
6 7 8
9
10