Comparing two or more models fit to a common data. Also known as "model selection".

learn more… | top users | synonyms

1
vote
0answers
23 views

Effect size for comparison of two nested models

I want to calculate an effect size for the difference of two structural equation models (model 1 and 2). Model 2 is nested in model 1. For each model I get a $\chi^2$ as measure of fit. Whether the ...
1
vote
0answers
13 views

AIC (QIC) - comparision of models?

I am comparing models created by geeglm function in R software. Does QIC of a model need to be lower than QIC of a null model? (in geeglm, geepack, R)
4
votes
3answers
270 views

Is overfitted model with higher AUC on test sample better than not overfitted one

i am participating in a challange in which I have created a model that performs 70% AUC on train set and 70% AUC on hold-out test set. The other participant has created a model that performs 96% AUC ...
2
votes
1answer
48 views

Over-parameterization in Bayesian Hierarchical Model

Can someone explain the influence of adding parameters to a Bayesian model? I have read from Kruschke that Bayesian analysis 'accounts' for model complexity by way of multiple priors, however I don't ...
0
votes
1answer
41 views

What statistical test to use - comparing models?

Say, I have created 3 models that evaluates the amount of harmful pollution. It uses a variety of factors (location, time of year, etc.) to determine this. I have a bunch of data for 10+ Cities from ...
3
votes
0answers
36 views

Incorporate an external estimate of probability as predictor in a logistic regression model

I am predicting a binary outcome (e.g., credit default) with logistic regression. For each observation, in addition to my own observed predictors, I have obtained a probability estimate from an ...
1
vote
1answer
70 views

Interpreting output of analysis of deviance table from anova() model comparison

0 down vote favorite I have a large multivariate abundance data and I am interested in comparing multiple models that fit different combinations of three categorical predictor variables to my species ...
1
vote
1answer
28 views

Comparing confusion matrices from model fitting

Using R, I got a bunch of confusion matrices from some model fitting. I'm trying to choose the best model by looking at their confusion matrix. Not an easy task. My current method of comparison is to ...
1
vote
0answers
19 views

Comparison of decision trees in R cran

Is it possible to compare two (or more) models of decision tree obtained with ctree on different data? In particular I would like a statistical test that compares two decision trees (builded with the ...
0
votes
0answers
59 views

Data normalization/standarization and comparing

I have a question related to data comparing. First of all, my dataset is composed by cities of the world. In this cities, we have a maximum of 24 tags that indicate what these cities are best for. ...
0
votes
0answers
26 views

An interesting two model comparison metric

My colleagues at work use an interesting metric to compare two predictive models that I have never seen before. It can be used for both classification and regression. But we consider classification ...
0
votes
0answers
4 views

Comparing set of counts with different n sample and many zeros.

I had a look of may treats and find different pieces here and there but I cant figure out how to do this "simple" comparison. I have 2 counts (obtained with different methods but assuming from same ...
1
vote
1answer
29 views

Using DIC for model selection: (in)valid comparisons

Imagine I have the following 7 glmm models where $b_1$ through $b_3$ are fixed effects. $M_1 = y \sim b_1 \times b_2 \times b_3$ $M_2 = y \sim b_2 \times b_3$ $M_3 = y \sim b_1 \times b_3$ $M_4 = ...
0
votes
1answer
17 views

Fitting and comparing distributions based on diverse summary statistics

I have a bunch of samples, about 35, drawn from a fat-tailed distribution. I think it is reasonable to assume that the samples are all drawn from distributions from the same distributional family, ...
0
votes
1answer
28 views

Need a comparison method to compare original data values and data values derived from a method

I have a gene expression dataset for which I have the original data values and I have put this dataset through a few methods to test their accuracy of prediction. I need to compare the results from ...
1
vote
0answers
28 views

How to compare different models in H2O? [closed]

I am new to H2O and I have just used in learning context. I am trying to figure out how to compare different models in order to chose the one with the best in a binary classification scenario. I have ...
0
votes
0answers
7 views

Compare GLM with different dependent variables (that have not the same variance)

I would like to know if it is possible to compare the effects of one predictor on different response variables that follow e.g. a Poisson distribution. I have already read this topic with excellent ...
0
votes
0answers
14 views

Model comparison using lower bound from variational approximation

I applied variational approximation for probit regression model and got the lower bound for the log marginal likelihood. When I compare models with different covariates using lower bound, I found that ...
0
votes
0answers
9 views

Difference between weighting and replicating observations in linear regression

I have a model in which each case is summary statistic of many observations. I am using a mixed effect model (lmer() in R) for prediction and I thought to give ...
2
votes
1answer
32 views

How to compare a Log-Log regression models with a Support Vectors Machine model (SVM)?

I have developed a log-log model which gives me a rmse of 0.1. I want to compare the results with a SVM model. In the SVM i didn't initially use the log transformed variables. RMSE from the non-...
0
votes
1answer
29 views

Comparing and evaluating win probabilities in sports from different settings

Background I'm trying to predict the probability that the home teams wins a certain sports game, for each minute of the game. Taking these win probabilities together produces a nice visual of the ...
5
votes
1answer
70 views

Bayesian model comparison in high school

I teach physics to high-school students, and I would like my students to conduct a rudimentary Bayesian model comparison for data from their experiments. I figured out a way for them to do so (see ...
2
votes
1answer
24 views

I want to compare the risks of smoking two different cigarettes

I want to compare the risks of smoking one brand of cigarette versus another. Each cigarette when smoked produces a list of toxic chemicals. I can measure each chemical and come up mean and standard ...
0
votes
0answers
16 views

range of possible values and model complexity

I have a prediction model where the likelihood is a function of two components $(A,B)$ $L \sim \bf{A}^\alpha + \bf{B}^\beta$ , where both $\bf{A}$ and $\bf{B}$ are $n\times n$ matrices ($n$ is the ...
3
votes
0answers
113 views

Am I calculating the Anderson-Darling statistic correctly for comparing PDFs?

I want to compute the Anderson-Darling statistic to use as a goodness of fit test for some PDF distributions. I am using R. I want to compare some models to the observations, which are my baseline. ...
2
votes
1answer
31 views

How to compare predictive power of sports odds

For my MSc Thesis I would like to compare the predictive power of classic bookmakers on the one hand and a betting exchange on the other. I have a lot of data on both types of betting, mostly on ...
0
votes
1answer
24 views

How compare more than two methods after cross validation

I am extremely rusty at statistics so maybe this is an 'easy' question but I haven't been able to find what I would like to know with a simple search. I am using now a Wilcoxon signed rank sum test ...
0
votes
0answers
10 views

Compare inferred parameters

Given one set of data I fit the same model in two different ways. I now have the inferred values for the parameters and the standard error. How can I test if they are statistical different? I have ...
1
vote
0answers
13 views

Comparing count groups/hisotgrams with different bins

I'm looking at clustering techinques for which I have a ground truth. From the ground truth I can extract the cardinality (size) of clusters and I would like to test how well different methods can ...
0
votes
0answers
20 views

Scenario of how to compare models

I have a binary classification problem where the distribution of classes is skewed. I've already trained some scoring models with logistic regression. Now I would like to compare them. How to do this ...
0
votes
0answers
15 views

Comparing binary models. ROC convex hull method

I have a binary classification problem where the distribution of classes is skewed. This is what I am doing right now: I train some logistic regression scoring models (different features, ...
0
votes
0answers
19 views

How to compare two factors as predictors of a continuous variable

I have a question but I haven't found the right terms to ask it, so I'll try to explain the problem in abstract terms. I have a few predictor variables X (continuous) and two factors, say, A and B. I ...
1
vote
0answers
30 views

Bayes factor for effect size

I need to construct Bayes Factor for testing: $$ H_0: \quad 0 = \mu_t - \mu_c $$ $$ H_1: \quad 0 \neq \mu_t - \mu_c $$ and my prior knowledge is modeled by distribution of effect size: $ \delta = \...
0
votes
0answers
27 views

Comparison of different forecasting models

I have a time series data of 1000 points (in %) for each of the different machines. I tried different forecasting techniques to make a one step prediction. The goal is to find out one common ...
0
votes
0answers
8 views

Compare whether two models with different output units agree with each other

I am trying to model whether a location is good for cultivating a crop. I have 7000 locations and I constructed two different models: first model gives me the probability of presence of a crop (...
0
votes
0answers
45 views

Using the 'product space method' to test the evidence for hypotheses of random effects in hierarchial Bayesian models

I am exploring the use of product space methods, coded in JAGS within R, for Bayesian model selection/comparison. I am particularly interested in using this method to test hypotheses about random ...
0
votes
2answers
48 views

Choosing between two regression models

I have two regression models: Model A: 569 observations 13 predictors Model B: 569 observations 14 predictors Model diagnostics are: R^2 value of Model A < R^2 value of Model B F-statistic ...
0
votes
0answers
10 views

Can I use ANOVA to compare two partial least square model?

When comparing two linear regression, we could use ANOVA as below. anova(model1,model2,test="Chisq") But when comparing two partial least square model, I could ...
0
votes
0answers
12 views

Comparing single-value and probabilistic online predictions

Let $\mathbf{x}^{(T)} = \{ x_1, x_2, \cdots, x_T \}$ ($x_t \in \mathbb{R}^d$) and $\mathbf{y}^{(T)} = \{ y_1, y_2, \cdots, y_T \}$ ($y_t \in \mathbb{R}$) be two time series. If I had a "single-value",...
0
votes
0answers
16 views

Correct evaluation/ comparison between undercomplete and overcomplete representations

Suppose I'm performing Unsupervised Feature Learning method to learn a representation of the data that is under-complete (e.g. 100 features) and use another algorithm to learn an over-complete ...
3
votes
1answer
81 views

Different results using Brier score and Logarithmic scoring rule

Let $X_i\sim B(\pi_i), \text{for }i=1,2,\cdots,n$. I have two models and I want to compare which of them forecast better. Model 1: Estimates the parameters with maximum likelihood. Model 2: ...
1
vote
1answer
48 views

Bayes factor from posterior odds

I tried to answer my own question Comparing two Bayesian models under disjoint prior supports using MCMC. Here is my intent. I am not confident in what I wrote so prefer to post it as a question : Is ...
1
vote
0answers
46 views

Are Bayes factors practically applicable?

According to Bishop's Pattern Recognition and Machine Learning on page 164, on average the Bayes factor will always favour the correct model. Given this, how can we use the Bayes factor in ...
0
votes
2answers
108 views

AIC/BIC and data transformation

Can you use AIC/BIC to compare models on untransformed data with models on transformed data (such as log, inverse hyperbolic sine, etc.)? I.e. if a model using logged data gives an AIC = 53.62 and a ...
1
vote
0answers
81 views

AIC comparison for models without maximum likelihood

AIC is a popular model comparison measure (despite of its potential shortcomings). I am wondering whether it is legit to compare AIC (or Akaike weights) for models that were fitted without requiring ...
2
votes
2answers
46 views

Comparing two Bayesian models under disjoint prior supports using MCMC

I have a Bayesian model involving three parameters $\theta_1$,$\theta_2$ and $\theta_3$. Experts think that $\theta_1 > \theta_2 > \theta_3$. So I would like to test the submodel $M_0$ ...
0
votes
0answers
51 views

Calculate AIC or Anova from an RJags model

I am fitting multivariate linear regressions with RJags (I have to do it with an mcmc because I'm taking all errors into account). I want to know between two polynomials which one fits better my data....
3
votes
1answer
86 views

Can I use linear regression coefficients in a logistic regression model?

This is a hacky question. What I'm hoping for is answers that identify and explain the problems and possible solutions. I'm dealing with some code that is already in production that uses a logistic ...
1
vote
0answers
25 views

Method to quantify differences between prediction and outcome for factual and simulated sample

I have vector with probability values a and binary outcome b: ...
0
votes
0answers
13 views

Linear regression with different samples of data: addressing compatibility and parameter estimation

I have two samples of data (call 'em black and red), spread like in figure. I ran a linear regression over both samples. I understand that they cannot come from the same joint pdf f(x,y), and that a ...