Comparing two or more models fit to a common data. Also known as "model selection".

learn more… | top users | synonyms

0
votes
0answers
6 views

Correct evaluation/ comparison between undercomplete and overcomplete representations

Suppose I'm performing Unsupervised Feature Learning method to learn a representation of the data that is under-complete (e.g. 100 features) and use another algorithm to learn an over-complete ...
2
votes
1answer
26 views

Different results using Brier score and Logarithmic scoring rule

Let $X_i\sim B(\pi_i), \text{for }i=1,2,\cdots,n$. I have two models and I want to compare which of them forecast better. Model 1: Estimates the parameters with maximum likelihood. Model 2: ...
0
votes
0answers
23 views

Bayes factor from posterior odds

I tried to answer my own question Comparing two Bayesian models under disjoint prior supports using MCMC. Here is my intent. I am not confident in what I wrote so prefer to post it as a question : Is ...
1
vote
0answers
29 views

Are Bayes factors practically applicable?

According to Bishop's Pattern Recognition and Machine Learning on page 164, on average the Bayes factor will always favour the correct model. Given this, how can we use the Bayes factor in ...
0
votes
2answers
47 views

AIC/BIC and data transformation

Can you use AIC/BIC to compare models on untransformed data with models on transformed data (such as log, inverse hyperbolic sine, etc.)? I.e. if a model using logged data gives an AIC = 53.62 and a ...
1
vote
0answers
77 views

AIC comparison for models without maximum likelihood

AIC is a popular model comparison measure (despite of its potential shortcomings). I am wondering whether it is legit to compare AIC (or Akaike weights) for models that were fitted without requiring ...
0
votes
1answer
26 views

Comparing two Bayesian models under disjoint prior supports using MCMC

I have a Bayesian model involving three parameters $\theta_1$,$\theta_2$ and $\theta_3$. Experts think that $\theta_1 > \theta_2 > \theta_3$. So I would like to test the submodel $M_0$ ...
0
votes
0answers
26 views

Calculate AIC or Anova from an RJags model

I am fitting multivariate linear regressions with RJags (I have to do it with an mcmc because I'm taking all errors into account). I want to know between two polynomials which one fits better my ...
3
votes
1answer
77 views

Can I use linear regression coefficients in a logistic regression model?

This is a hacky question. What I'm hoping for is answers that identify and explain the problems and possible solutions. I'm dealing with some code that is already in production that uses a logistic ...
1
vote
0answers
19 views

Method to quantify differences between prediction and outcome for factual and simulated sample

I have vector with probability values a and binary outcome b: ...
0
votes
0answers
12 views

How do we characterize model richness under regularization?

Say we have N parameters in a linear regression model. Often, we add priors to say we expect those parameters to be 0, and then fit to some data points. (I've seen this used with "infinite" parameter ...
0
votes
0answers
9 views

Linear regression with different samples of data: addressing compatibility and parameter estimation

I have two samples of data (call 'em black and red), spread like in figure. I ran a linear regression over both samples. I understand that they cannot come from the same joint pdf f(x,y), and that a ...
0
votes
0answers
8 views

Compare two groups fitted with different models

I have two group means that "behave" in different ways. Group A increases and reaches a "plateau-like" value. Group B stays almost steady. I have used a polinomic fit (cubic) for group A and linear ...
0
votes
0answers
17 views

Maximum likelihood estimator for a variable and its reconstruction

I have a spatial variable $X_1$ observed at time $t_1$ and its original form $X_0$ observed at time $t_0$ (I don't have any info between these two stages). I want to reconstruct $X_1$ on the basis of ...
1
vote
0answers
16 views

Deriving the F-test for a complex comparison

I would like to clarify a passage from Maxwell & Delaney, Ch.4 Extension in which the authors explain how to perform an F-test for a complex comparison. In what follows, the coefficients $c_j$ ...
2
votes
1answer
39 views

Can we apply BIC to select between models in which the parameters are estimated by MCMC

I never considered this problem before. Assume we have two models, $A$ and $B$, we estimate the parameter in $A$ and $B$ respectively by sample mean. Two questions: We can always use the means of ...
1
vote
0answers
29 views

Best practice methods to compare 2 regression models?

I have two regression models that differ only by one thing: model 1 has one of the explanatory variables linear while model 2 takes the natural logarithm of it. I would like to know, what are the ...
1
vote
0answers
33 views

How to compare two different rankings of same set of objects?

I have $N$ objects $A_1,\dots,A_N$. I have two algorithms that assign two different scores to each of this objects. Thus sorting on this scores gives me two different rankings of these same objects. ...
0
votes
0answers
16 views

Compare Logistic regression with a generic model

I've a large dataset of $x_i$ quantitative and $y_i$ categorical (0,1). I fitted them using a Logistic regression, so the output produced is a boolean vector $y^{logit}_i$. Then I've also an ...
3
votes
1answer
66 views

Bayesian modeling and FDR correction

Question Assume that we have 2 experiments, each of which yields $N$ observations. We label them $(c_1^i,\ldots,c_N^i)$ with $i=1,2$. Each observation is a positive integer, and can be modeled as a ...
0
votes
0answers
6 views

Comparing two models for weighted binary data

What's a good method for comparing a poisson regression with weigted observations to another model, also weighted? Would a scoring rule work?
0
votes
1answer
516 views

R - Lift chart analysis & classification tree - ROCR

I am trying to compare several models of classification tree using the ROCR package however, the x-axis in this package ...
1
vote
1answer
114 views

WEKA Experimenter: Statistically significant results appear counterintuitive

In WEKA Experimenter I have a problem that seems counterintuitive and I would like to know whether I correctly interpret the results. My experiment specifics: Type of experiment: Regression ...
1
vote
0answers
48 views

BIC difference for model selection when models have different (and correlated) predictors

I have a binomial dependent variable Y and two main IVs: A is categorical (5 non ordered levels) and B is continuous. A and B are collinear (I tested the effect of A on B with a linear model, and it ...
0
votes
0answers
22 views

Fusion of two model reduced sensing systems / Comparison of two models

General context: I have a computer vision problem where I take an image sample then: Analyse* it (details omitted) to obtain a vector of values Run PCA (Principle Component Analysis) on it to ...
0
votes
0answers
48 views

LMM: Model-comparison and evaluation

My problem is that the residuals of two models (that differ significantly) are not different, which confuses me. If two models are different, then surely we can expect the residuals from one of the ...
1
vote
2answers
85 views

Comparing individual predictions across two models

I want to find patterns in the way that two models, Model A and Model B, perform on my dataset. What are some good ways to compare the individual predictions of the two models for $N$ classes? If any ...
0
votes
1answer
36 views

Compare two sets of linear mixed effects models

I have two questions that (i think) should be solved in the same way. I'm using the lme4 package in R to analyze how well a set of subjective ratings (from 12 participants) tracks an objective ...
0
votes
0answers
20 views

Method comparisons for trees and forests

I am looking to compare various types of models for a dataset, in order to determine which one is the most suitable. They will all be a form of decision tree, from the basic tree to random forests, ...
0
votes
0answers
19 views

Taking differences, indexes or levels in regression analysis?

I am new to econometrics and regression analysis and I am trying to make a decision on which path to follow on my regression. I have cross sectional data, and I want to estimate the impact of ...
1
vote
0answers
18 views

Compare logistic models on same customers

I have two logistic models on the same set of customers in telecom. 1. Propensity to convert from fixed line to prepaid line 2. Propensity to convert from fixed line to post-paid line. I have to ...
0
votes
0answers
18 views

Comparing a regression in two scenarios

I am looking for some answers regarding regression. I am running a regression in scenario 1 and then same construct measured for a scenario 2, the scenario is different but the constructs are the ...
0
votes
1answer
117 views

Equivalent measure to Matthews correlation coefficient, MCC, for multiclass classification

Thanks in advance for the help. MCC gives a measure of the quality of a binary classifier. I'm looking for a similar measure that can be used for a multi-class classifier. Ultimately what I would ...
1
vote
0answers
81 views

Interpretation of cross validation results when comparing models

I'm trying to solve a bio-medical image segmentation problem using a binary classifier and then a spatial smoothing (assuming continuous regions). I have: Training set of 10 3D scans, a total of ~30 ...
1
vote
1answer
30 views

Comparison of time series models

I'm trying to create a model for a series $X = \{X_1, X_2, ...\}$. I don't assume that the $X_i$ are identical distributed nor that they are independent but at least that they have something in common ...
0
votes
0answers
34 views

AIC for multiple nonlinear regression models

How do we got about using AIC for multiple nonlinear regression models ? For example: If i have a dataset with N instances, and they can be explained by a collection of 3 models where each model has ...
1
vote
1answer
116 views

What is the use of splitting dataset into training/test prior to cross-validation?

I've occasionally seen people advocate splitting the full dataset into training/test (typically a 70/30 or 80/20 split) and then running CV on the training set. I don't fully understand the point of ...
0
votes
0answers
110 views

How to interpret the R output of anova.coxph

The likelihood ratio chi^2 test is the gold standard for comparing nested Cox models. However I am not sure how to interpret the R output. ...
2
votes
1answer
83 views

How to determine similarity between histograms? (Which metric to use?!)

Hello Stack Exchange community, I'm relatively confused on what to do in this scenario. I am running an experiment that retrieves data of interest and stores it as an array. Experimental conditions ...
0
votes
1answer
28 views

Testing the utility of adding a continuous variable to a nonlinear regression.

Let’s say I have the hypothesis that soil fertility affects the relationship between weed biomass and crop biomass. One way to go about testing that hypothesis might be to model the relationship ...
0
votes
0answers
21 views

Appropriate statistic for showing advantage of a nonlinear model over its linear counterpart?

I am interested in demonstrating the utility of nonlinear model compared to its linear alternative. Since the outcome I am using is binary, the usual approach I can think of may be compare the ...
0
votes
0answers
28 views

Comparing models on two target variables

I have a dataset with two binary target variables $Y_a$ and $Y_b$. I can build two classification tree models: $\hat{Y}_a = M_a(X)$ $\hat{Y}_b = M_b(X)$ Questions: Is there a way to compare the ...
0
votes
0answers
58 views

reporting mixed effects linear regression: t statistic or model comparison?

We have a one factor three level repeated measure experiment with ratio data (reaction times). We fit a mixed effects linear model using lmer (in fact lmerTest) - maximal, with subjects and items as ...
3
votes
0answers
70 views

Approximating the marginal likelihood in Bayesian Model Comparison

Given some data $y$, my interest centers around a collection of models $\{\mathcal{M}_1,\mathcal{M}_2,\cdots,\mathcal{M}_L\}$ representing competing hypotheses about $y$. Each model $\mathcal{M}_l$ ...
1
vote
0answers
75 views

model comparison: chi2 vs. AIC (and BIC)

I've been using the approach from Appendix A from this paper. It states that if I have two fits with $\chi^2$ values $\chi^2_1(\nu_1)$ and $\chi^2_2(\nu_2)$ ($\nu$ - degree of freedom), then their ...
0
votes
0answers
11 views

Comparing logistic regression models with different predictors [duplicate]

What measure do I use to compare two logistic regression models with different predictors but the same response? y ~ x y ~ z I've used lrtest and anova before ...
0
votes
0answers
107 views

Comparing models using different sample sizes from the same longitudinal cohort

Epidemiological study: All of the 5 models I am comparing are derived by data from same longitudinal cohort. Each model contains the same IV, DV, and covariates. The difference between each model ...
2
votes
0answers
47 views

Comparing logistic regression models with paired data, population-averaged estimates and robust standard errors

I would like to ask if any of you have suggestions for comparing logistic regression models with paired data (3 observations/ID code), population-averaged estimates and robust standard errors (I am ...
0
votes
0answers
23 views

How can test corpora (with different sizes) be compared to each other via accuracy?

I need an advice regarding a text-classifiction scenario. I use a simple accuracy measure defined as #(correctly classified classes in a corpus X) / #(all docs in X). Let A, B and C be three ...
4
votes
0answers
79 views

F-test to determine whether more than two sets of data differ

Here is the context for my question: I understand that you can fit the same model to two different datasets separately and then fit the model to the datasets pooled together as a way to discern ...