Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

0
votes
0answers
10 views

How to do regression model selection if dummy variables are involved?

Original post on stackoverflow: http://stackoverflow.com/questions/28773153/how-to-do-regression-model-selection-if-dummy-variables-are-involved I am trying to do a logistic regression analysis in R ...
0
votes
0answers
6 views

Text classification: Dealing with potentially false classified instances

My goal is to complete a supervised text classification task with R. There are several classes, of which some of the class counts will be relevant for a subsequent analysis. I have already tried a few ...
1
vote
0answers
18 views

Device Comparision: Correlated or uncorrelated measurements

Background: I want to compare two devices measuring a certain characteristic on a subject. Thereto, each subject is measured once with device A and once with device B. It needs to be assumed that ...
0
votes
0answers
8 views

How to apply AIC to a situation where the mean of a multivariate normal is a 0-1 d-dimensional vector with exactly k 1's

I am trying to apply AIC to estimate mean in the following case: Let us consider that I have $n$ random variables $X_1, \ldots, X_n$, drawn i.i.d. from a normal distribution of mean $\mu\in\{0,1\}^d$ ...
0
votes
0answers
19 views

Model Stacking algorithm

I'm trying the stacking method to see if it improves my results, but before using some R package, i decided to code it by myself. Here's a pseudocode of what i'm doing: ...
1
vote
1answer
17 views

Model selection: can I compare the AIC from models of count data between linear and poisson models?

I am modeling count data (with offset / exposure parameter). My modeling strategy is use of a Poisson model and a negative binomial regression model. I compare model AICs, which are about -760 for my ...
0
votes
0answers
30 views

AIC versus cross validation in time series

I am interested in model selection in a time series setting. For concreteness, suppose I want to select an ARMA model from a pool of ARMA models with different lag orders. The ultimate intent is ...
0
votes
0answers
7 views

Heckman Probit Model the number of explanatory variable in selection model?

I run a Heckman Probit model which is sometimes called as Heckit. It consists two parts like this: |1| Y X1 X2 X3, |2| select(Y2 X1 X2 X3) Y covers Y2 but not vice versa. The question is whether i ...
7
votes
2answers
151 views

Is using correlation matrix to select predictors for regression correct?

A few days ago, a psychologist-researcher of mine told me about his method to select variables to linear regression model. I guess it's not good, but I need to ask someone else to make sure. The ...
1
vote
0answers
32 views

Strictly positive response in regression: what should my “default” model be?

For unbounded continuous responses, Gaussian errors are the analyst's default model for many reasons, one of them being that their ML estimate coincides with the OLS estimate that has many desirable ...
1
vote
1answer
25 views

Proper variable selection: Use only training data or full data?

I'm going through the lab exercises in "Introduction to Statistical Learning" and am having difficulty understanding the proper way to do best subset selection. The book is available here ...
0
votes
0answers
9 views

VAR model selection for forecasting one variable

Suppose I have a VAR model for variables $x_1$ through $x_K$. I will use the model to forecast $x_1$ a few steps ahead and will do this iteratively rather than directly. I am not interested in ...
1
vote
0answers
15 views

Are wrong standard errors a problem if using information theoretic model selection?

In linear regression, if the assumptions of normally distributed residuals and homogenous residuals are broken, incorrect standard errors can be calculated. This can lead to some predictors appearing ...
0
votes
0answers
16 views

Model choice for discrete data with R

I'm keen to model the effect (impact) of variable i.e. var1 and its effect (causation inference?) on var2 given the score measure. Sample data (variation of data I have got) ...
0
votes
0answers
9 views

ensemble model for SVM

I did a nested 5-cv and the resulting models are unstable (high variance among the hyper parameters C and gamma of SVM). So, I don't know how to choose C and gamma for the "final" model. I read that, ...
1
vote
1answer
32 views

Goodness of fit between actual values and non-linear model

I was wanting to get a goodness of fit similar to R^2 for a model I'm evaluating. The output of the model is one of 8 numbers based on environmental characteristics. This is not a linear model, so ...
1
vote
0answers
28 views

GLMM with 2 insignificant variables has lower AIC or BIC compared to same model without those variables…?

I am having a hard time understanding what's going on in with my model selection, and why a model with two insignificant variables is getting chosen as the "best model" over a model without those two ...
2
votes
0answers
51 views

Model selection

I have a small dataset of 37 observations with students' performance on both cognitive tests (5) and professional tests (6). My goal is to predict professional tests (DV) with cognitive tests(IV). To ...
1
vote
0answers
24 views

How to deal with many NAs in dataset when comparing several models?

My task is to compare different logistic regression models to examine which theory better explains the data we have collected. In doing so, I have a set of IVs with a considerable amount of missing ...
0
votes
0answers
20 views

Model Selection Problem

I am asking if there already exist approaches and researches on the following topic. Imagine there are 10 stores and in 3 stores labeled training data was available, so I built 3 classification ...
0
votes
1answer
33 views

Comparison between a multilevel and an unpooled model

Suppose we have fitted two models: a multilevel model and an unpooled model: m1=lmer(y~x+(1|group)) m2=lm(y~x+factor(group)-1) How can I understand which ...
0
votes
1answer
16 views

How to identify the ARCH and GARCH lag lenght in DCC GARCH Model

I just follow the Stata manual for DCC GARCH model. This model contains ARCH(1) and GARCH(1) terms. But my question is, on what basis and how we can can select ...
0
votes
1answer
45 views

Leave-One-Subject-Out cv method

I would like to use a Leave-One-Subject-Out cv on my datasets (I have dataset including 38, 15, 10 participants, respectively). I don't know the hyperparamenters C and gamma of my SVM so I have to ...
6
votes
1answer
90 views

Statistics for model selection and model evaluation

In his Dynamic Econometrics, David F. Hendry issues the following advice: When a 'test' statistic is cited, you must ask 'Was that a selection criterion statistic or a genuine test statistic?' ...
2
votes
1answer
71 views

Cross-validation and logistic regression

I'm interested in building a set of candidate models in R for an analysis using logistic regression. Once I build the set of candidate models and evaluate their fit to the data using AICc (...
2
votes
1answer
41 views

How does step function selects best linear Models which includes polynomial effects and interaction effects in R?

I try to find "best" linear models with continuous and categorical covariables with Interaction Effect by BIC. The continuous covariables should have a quadratic effect on the response variable. ...
3
votes
2answers
65 views

Generate fake data consistent with adjusted R^2 pattern

Is it possible to specify a vector of adjusted $R^2$ values (or any other measure like AIC, BIC, $C_p$) for the set of all possible models in a data set, and then generate data that is consistent with ...
1
vote
1answer
33 views

Is there a default parameter choice for the spike-and-slab prior?

In the spike-and-slab prior, one needs to specify $h_{0j} = P(\beta_j=0)$, which demonstrates our prior belief about how likely $\beta_j$ to be an important predictor. Is there a default choice for ...
0
votes
1answer
24 views

Holdout set for image task

I need to validate whether one or two templates/shapes are present in an image. Fitting two templates has a better maximum likelihood then fitting one template which is a clear symptom of overfitting. ...
0
votes
1answer
76 views

How to use LASSO to select glm model gaussian

I have a small sample size n<20. I want to find which combination of 8 variables better predict y. I was using a stepAICc but it is suggested to away stepwise model selection. I have tried lars ...
0
votes
1answer
39 views

is there a way to plot best glm model in model selection

I have run this glm model y~poly(xa,2)+poly(xb,2)+... Then have found the best fitting model using AICc. The best fitting model has a subset of the ...
0
votes
0answers
32 views

Model selection in nonlinear fitting

Learning curves are fitted with multiple trendlines (exponential, power, logarithm). The fitting is performed by the Levenberg–Marquardt algorithm. So far so good. The question is, how to select the ...
2
votes
2answers
115 views

Using LASSO for variable selection, then using Logit

I know this would muddy the statistical inference, but I am really only concerned with getting as close to an accurate model as I can. I have a dichotomous outcome variable, with a large set of ...
2
votes
1answer
40 views

How is the Akaike Information Criteron applied for model with large number of predictors?

I am reading a paper (details not very relevant) which assumes that the market cost $C$ of a trade is related to $N$ predictors $X_1,\dots,X_N$ (page 25) through a linear relationship $$C = \beta_0 + ...
23
votes
3answers
2k views

Is p-value essentially useless and dangerous to use?

First are some background information. This article "The Odds, Continually Updated" from NY Times happened to catch my attention. To be short, it states that [Bayesian statistics] is proving ...
0
votes
0answers
10 views
0
votes
0answers
10 views

Does it make sense to report equally-fit, more complex, model, if it fits better a theory?

I have two (logistic) regression models for which the deviance is not significantly distinct (p = 0.7). One of them has education, gender and age explaining variable Y. In the other, I have added a ...
6
votes
3answers
192 views

Analyse ACF and PACF plots

I want to see if I am on the right track analysing my ACF and PACF plots: Background: (Reff: Philip Hans Franses, 1998) As both ACF and PACF show significant values, I assume that an ARMA-model ...
1
vote
1answer
70 views

The best model of an AICc-based model selection on a very small sample has an high number of predictors: does it make any sense?

I'm working with a very small sample size (N=14) and I'm using AICc to identify the most parsimonious model using a large number of possible predictors. Unexpectedly the best model has six predictors! ...
0
votes
0answers
28 views

Assesing the explanatory power of predictors, interactions and combination of terms

I have a model with 5 basic predictors and all interactions between the predictors themselves. Something like (I'm simplifying here, in reality I have many more variables): ...
0
votes
1answer
34 views

Comparisons of confirmatory and exploratory model in IRT

I am trying to decide whether a theoretically derived (i.e. confirmatory) IRT model fits the data better than some parsimonious (i.e. exploratory) IRT model. Specifically, I have 14 binary indicators ...
4
votes
2answers
93 views

Is it possible to compare the parsimony of models with the same number of parameters and explanatory variables?

Parsimony is often defined as the minimisation of unnecessary parameters or explanatory variables in a model. But models also have structure - functional forms that can change. Between two models that ...
2
votes
0answers
13 views

Half-Normal Plot of Coefficients from Binary Factorial Experiments

After I wrote this all up I debated whether or not I should post it because I think I know the answer to this question (after looking at the two models I'd end up with), but since I don't really know ...
0
votes
0answers
25 views

Modeling of Multivariate Data

Suppose I have a multivariate data set. For the sake of example, lets say that the dimension of my data set is $p=7$ and I have a matrix which contains samples of this multivariate data set. Now ...
1
vote
1answer
31 views

K-fold validation, how to use MSE and STD for model selection

When using K-fold validation for model selection I'm wondering what's the best approach to select a model using both the mean square error (MSE) and the standard deviation of errors among folds (STD). ...
0
votes
2answers
68 views

p value vs prediction error

In a lot of fields (like medicine) to check if a variable is related to an output is controlled if the p-value of that variable in a regression model is significant. For example: ...
1
vote
1answer
54 views

Marginal Likelihood in PYMC

I am using the PYMC toolbox in python in order to carry out a model selection problem using MCMC. What I would like to have for each model is the marginal log-likelihood (i.e. model evidence). The ...
1
vote
0answers
14 views

Testing sequentially nested models

Assume I have a simple model with (at least) four parameters $\beta_1, \beta_2, \beta_3, \beta_4$. If I would want to test $H_o: \beta_1 = \beta_2 \& \beta_3 = \beta_4$ by using the likelihood ...
0
votes
0answers
23 views

Model-selection and interactions: Glmnet dropping underlying terms

I have a dataset and want to do an Ancova with several explanatory variables (several factors and two covariates) and their interactions. I want to select the best model using glmnet and lasso ...
1
vote
3answers
39 views

How to compare 2 predictive models where one uses predictor with missing values

I am developing a model to predict y from a dataset (N=20,000) that contains x1, x2. Say I ...