Comparing two or more models fit to a common data. Also known as "model selection".

learn more… | top users | synonyms

0
votes
0answers
12 views

Model comparison using lower bound from variational approximation

I applied variational approximation for probit regression model and got the lower bound for the log marginal likelihood. When I compare models with different covariates using lower bound, I found that ...
0
votes
0answers
9 views

Difference between weighting and replicating observations in linear regression

I have a model in which each case is summary statistic of many observations. I am using a mixed effect model (lmer() in R) for prediction and I thought to give ...
2
votes
1answer
29 views

How to compare a Log-Log regression models with a Support Vectors Machine model (SVM)?

I have developed a log-log model which gives me a rmse of 0.1. I want to compare the results with a SVM model. In the SVM i didn't initially use the log transformed variables. RMSE from the ...
0
votes
1answer
25 views

Comparing and evaluating win probabilities in sports from different settings

Background I'm trying to predict the probability that the home teams wins a certain sports game, for each minute of the game. Taking these win probabilities together produces a nice visual of the ...
5
votes
1answer
69 views

Bayesian model comparison in high school

I teach physics to high-school students, and I would like my students to conduct a rudimentary Bayesian model comparison for data from their experiments. I figured out a way for them to do so (see ...
2
votes
1answer
24 views

I want to compare the risks of smoking two different cigarettes

I want to compare the risks of smoking one brand of cigarette versus another. Each cigarette when smoked produces a list of toxic chemicals. I can measure each chemical and come up mean and standard ...
0
votes
0answers
13 views

range of possible values and model complexity

I have a prediction model where the likelihood is a function of two components $(A,B)$ $L \sim \bf{A}^\alpha + \bf{B}^\beta$ , where both $\bf{A}$ and $\bf{B}$ are $n\times n$ matrices ($n$ is the ...
3
votes
0answers
110 views

Am I calculating the Anderson-Darling statistic correctly for comparing PDFs?

I want to compute the Anderson-Darling statistic to use as a goodness of fit test for some PDF distributions. I am using R. I want to compare some models to the observations, which are my baseline. ...
2
votes
1answer
19 views

How to compare predictive power of sports odds

For my MSc Thesis I would like to compare the predictive power of classic bookmakers on the one hand and a betting exchange on the other. I have a lot of data on both types of betting, mostly on ...
0
votes
1answer
24 views

How compare more than two methods after cross validation

I am extremely rusty at statistics so maybe this is an 'easy' question but I haven't been able to find what I would like to know with a simple search. I am using now a Wilcoxon signed rank sum test ...
0
votes
0answers
9 views

Compare inferred parameters

Given one set of data I fit the same model in two different ways. I now have the inferred values for the parameters and the standard error. How can I test if they are statistical different? I have ...
1
vote
0answers
13 views

Comparing count groups/hisotgrams with different bins

I'm looking at clustering techinques for which I have a ground truth. From the ground truth I can extract the cardinality (size) of clusters and I would like to test how well different methods can ...
0
votes
0answers
18 views

Scenario of how to compare models

I have a binary classification problem where the distribution of classes is skewed. I've already trained some scoring models with logistic regression. Now I would like to compare them. How to do this ...
0
votes
0answers
13 views

Comparing binary models. ROC convex hull method

I have a binary classification problem where the distribution of classes is skewed. This is what I am doing right now: I train some logistic regression scoring models (different features, ...
0
votes
0answers
18 views

How to compare two factors as predictors of a continuous variable

I have a question but I haven't found the right terms to ask it, so I'll try to explain the problem in abstract terms. I have a few predictor variables X (continuous) and two factors, say, A and B. I ...
1
vote
0answers
24 views

Bayes factor for effect size

I need to construct Bayes Factor for testing: $$ H_0: \quad 0 = \mu_t - \mu_c $$ $$ H_1: \quad 0 \neq \mu_t - \mu_c $$ and my prior knowledge is modeled by distribution of effect size: $ \delta = ...
0
votes
0answers
27 views

Comparison of different forecasting models

I have a time series data of 1000 points (in %) for each of the different machines. I tried different forecasting techniques to make a one step prediction. The goal is to find out one common ...
0
votes
0answers
6 views

Compare whether two models with different output units agree with each other

I am trying to model whether a location is good for cultivating a crop. I have 7000 locations and I constructed two different models: first model gives me the probability of presence of a crop ...
0
votes
0answers
35 views

Using the 'product space method' to test the evidence for hypotheses of random effects in hierarchial Bayesian models

I am exploring the use of product space methods, coded in JAGS within R, for Bayesian model selection/comparison. I am particularly interested in using this method to test hypotheses about random ...
0
votes
2answers
46 views

Choosing between two regression models

I have two regression models: Model A: 569 observations 13 predictors Model B: 569 observations 14 predictors Model diagnostics are: R^2 value of Model A < R^2 value of Model B F-statistic ...
0
votes
0answers
7 views

Can I use ANOVA to compare two partial least square model?

When comparing two linear regression, we could use ANOVA as below. anova(model1,model2,test="Chisq") But when comparing two partial least square model, I could ...
0
votes
0answers
12 views

Comparing single-value and probabilistic online predictions

Let $\mathbf{x}^{(T)} = \{ x_1, x_2, \cdots, x_T \}$ ($x_t \in \mathbb{R}^d$) and $\mathbf{y}^{(T)} = \{ y_1, y_2, \cdots, y_T \}$ ($y_t \in \mathbb{R}$) be two time series. If I had a ...
0
votes
0answers
15 views

Correct evaluation/ comparison between undercomplete and overcomplete representations

Suppose I'm performing Unsupervised Feature Learning method to learn a representation of the data that is under-complete (e.g. 100 features) and use another algorithm to learn an over-complete ...
3
votes
1answer
60 views

Different results using Brier score and Logarithmic scoring rule

Let $X_i\sim B(\pi_i), \text{for }i=1,2,\cdots,n$. I have two models and I want to compare which of them forecast better. Model 1: Estimates the parameters with maximum likelihood. Model 2: ...
1
vote
1answer
45 views

Bayes factor from posterior odds

I tried to answer my own question Comparing two Bayesian models under disjoint prior supports using MCMC. Here is my intent. I am not confident in what I wrote so prefer to post it as a question : Is ...
1
vote
0answers
38 views

Are Bayes factors practically applicable?

According to Bishop's Pattern Recognition and Machine Learning on page 164, on average the Bayes factor will always favour the correct model. Given this, how can we use the Bayes factor in ...
0
votes
2answers
83 views

AIC/BIC and data transformation

Can you use AIC/BIC to compare models on untransformed data with models on transformed data (such as log, inverse hyperbolic sine, etc.)? I.e. if a model using logged data gives an AIC = 53.62 and a ...
1
vote
0answers
80 views

AIC comparison for models without maximum likelihood

AIC is a popular model comparison measure (despite of its potential shortcomings). I am wondering whether it is legit to compare AIC (or Akaike weights) for models that were fitted without requiring ...
2
votes
2answers
46 views

Comparing two Bayesian models under disjoint prior supports using MCMC

I have a Bayesian model involving three parameters $\theta_1$,$\theta_2$ and $\theta_3$. Experts think that $\theta_1 > \theta_2 > \theta_3$. So I would like to test the submodel $M_0$ ...
0
votes
0answers
46 views

Calculate AIC or Anova from an RJags model

I am fitting multivariate linear regressions with RJags (I have to do it with an mcmc because I'm taking all errors into account). I want to know between two polynomials which one fits better my ...
3
votes
1answer
86 views

Can I use linear regression coefficients in a logistic regression model?

This is a hacky question. What I'm hoping for is answers that identify and explain the problems and possible solutions. I'm dealing with some code that is already in production that uses a logistic ...
1
vote
0answers
25 views

Method to quantify differences between prediction and outcome for factual and simulated sample

I have vector with probability values a and binary outcome b: ...
0
votes
0answers
13 views

Linear regression with different samples of data: addressing compatibility and parameter estimation

I have two samples of data (call 'em black and red), spread like in figure. I ran a linear regression over both samples. I understand that they cannot come from the same joint pdf f(x,y), and that a ...
0
votes
0answers
13 views

Compare two groups fitted with different models

I have two group means that "behave" in different ways. Group A increases and reaches a "plateau-like" value. Group B stays almost steady. I have used a polinomic fit (cubic) for group A and linear ...
0
votes
0answers
17 views

Maximum likelihood estimator for a variable and its reconstruction

I have a spatial variable $X_1$ observed at time $t_1$ and its original form $X_0$ observed at time $t_0$ (I don't have any info between these two stages). I want to reconstruct $X_1$ on the basis of ...
1
vote
0answers
18 views

Deriving the F-test for a complex comparison

I would like to clarify a passage from Maxwell & Delaney, Ch.4 Extension in which the authors explain how to perform an F-test for a complex comparison. In what follows, the coefficients $c_j$ ...
2
votes
1answer
54 views

Can we apply BIC to select between models in which the parameters are estimated by MCMC

I never considered this problem before. Assume we have two models, $A$ and $B$, we estimate the parameter in $A$ and $B$ respectively by sample mean. Two questions: We can always use the means of ...
1
vote
0answers
34 views

Best practice methods to compare 2 regression models?

I have two regression models that differ only by one thing: model 1 has one of the explanatory variables linear while model 2 takes the natural logarithm of it. I would like to know, what are the ...
1
vote
0answers
47 views

How to compare two different rankings of same set of objects?

I have $N$ objects $A_1,\dots,A_N$. I have two algorithms that assign two different scores to each of this objects. Thus sorting on this scores gives me two different rankings of these same objects. ...
0
votes
0answers
17 views

Compare Logistic regression with a generic model

I've a large dataset of $x_i$ quantitative and $y_i$ categorical (0,1). I fitted them using a Logistic regression, so the output produced is a boolean vector $y^{logit}_i$. Then I've also an ...
3
votes
1answer
81 views

Bayesian modeling and FDR correction

Question Assume that we have 2 experiments, each of which yields $N$ observations. We label them $(c_1^i,\ldots,c_N^i)$ with $i=1,2$. Each observation is a positive integer, and can be modeled as a ...
0
votes
0answers
7 views

Comparing two models for weighted binary data

What's a good method for comparing a poisson regression with weigted observations to another model, also weighted? Would a scoring rule work?
0
votes
1answer
870 views

R - Lift chart analysis & classification tree - ROCR

I am trying to compare several models of classification tree using the ROCR package however, the x-axis in this package ...
1
vote
0answers
225 views

AIC or ANOVA to compare models?

What are the relative merits of each approach, and which circumstances call for one rather than the other? To some extent I have a specific example in mind, which I've discussed here. In that example ...
1
vote
1answer
163 views

WEKA Experimenter: Statistically significant results appear counterintuitive

In WEKA Experimenter I have a problem that seems counterintuitive and I would like to know whether I correctly interpret the results. My experiment specifics: Type of experiment: Regression ...
1
vote
0answers
60 views

BIC difference for model selection when models have different (and correlated) predictors

I have a binomial dependent variable Y and two main IVs: A is categorical (5 non ordered levels) and B is continuous. A and B are collinear (I tested the effect of A on B with a linear model, and it ...
0
votes
0answers
22 views

Fusion of two model reduced sensing systems / Comparison of two models

General context: I have a computer vision problem where I take an image sample then: Analyse* it (details omitted) to obtain a vector of values Run PCA (Principle Component Analysis) on it to ...
0
votes
0answers
60 views

LMM: Model-comparison and evaluation

My problem is that the residuals of two models (that differ significantly) are not different, which confuses me. If two models are different, then surely we can expect the residuals from one of the ...
1
vote
2answers
98 views

Comparing individual predictions across two models

I want to find patterns in the way that two models, Model A and Model B, perform on my dataset. What are some good ways to compare the individual predictions of the two models for $N$ classes? If any ...
0
votes
1answer
42 views

Compare two sets of linear mixed effects models

I have two questions that (i think) should be solved in the same way. I'm using the lme4 package in R to analyze how well a set of subjective ratings (from 12 participants) tracks an objective ...