Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

0
votes
0answers
6 views

Sample size when fitting categorical survey data

I have a model which fits data from repeated surveys: at time $t$, a number $n_t$ respondents is asked a question and can give one of $K$ answers ($k=1, ..., K$). This is repeated $T$ times ($t = 1, ...
1
vote
1answer
16 views

Model selection and assumed parameters in models

Suppose there are four models: Model 1: $y = ax$ Model 2: $y = ax^2$ Model 3: $y = a\sqrt{x}$ Model 4: $y = ax^\theta$ Model 4 is the most complex model with two parameters (the others have ...
3
votes
1answer
34 views

Help constructing a simple regression model with a breakpoint

This is related to my questions here and here. I am still struggling with my model, so I am taking it back to basics. My assertion is simple, I believe that watershed runoff will have a different ...
0
votes
0answers
16 views

Proper Model Selection Randomized Block with Count Data

I have a data set on insect counts that looks like this: ...
4
votes
1answer
30 views

How to train a model when instead of a target we have a range where it is?

Often in machine learning we have a situation when target is numeric (real or integer). Each target comes with an associated input vector. The goal is to learn the mapping from the input vectors to ...
7
votes
1answer
69 views
+50

How to select the best fit without over-fitting data? Modelling a bimodal distribution with N normal functions, etc

I have an obviously bimodal distribution of values, which I seek to fit. The data can be fit well with either 2 normal functions (bimodal) or with 3 normal functions. Additionally, there is a ...
0
votes
1answer
18 views

Choose best samples to get strongest model

I am looking to create a model to best predict the presence of a particular species based on a number of habitat covariates. I have mapped out a couple of these covariates and am looking for a way to ...
1
vote
1answer
44 views

How to decide whether a variable belongs to a linear model?

I have a set of inputs $x$ and noisy outputs $y$. I think that either $$y = a_0 + a_1 x$$ or $$y = a_0 + a_1 x + a_2 x^2.$$ How can I determine which model was more likely to have generated the data? ...
1
vote
1answer
30 views

Does AIC require the residuals of the model to be normally distributed?

Does AIC require the residuals of the models to be compared to be normally distributed?
0
votes
0answers
12 views

Model selection (BIC / AIC) ordinal multilevel model containing a factor score and/or part of the factor

I am building a ordinal multilevel model (Stata 13.1; meologit-command). At this stage I am trying to conduct my model selection using the BIC / AIC. I estimated several models and now I need some ...
1
vote
1answer
42 views

GLM model selection using AICc with Tweedie distribution

I have two questions regarding use of Tweedie GLM in R. I am new in using this distribution and despite a thorough search on different forums, I could not find my answers. I am now running several ...
0
votes
0answers
13 views

Sampling from a set of non-nested models

Consider a collection $\mathcal{M}$ of $m$ different model classes $\mathcal{M} = \{M_1,\dots,M_m\}$, where each model class has a parameter set $\Theta_i$, $i=1,\dots,m$. The model classes are not ...
0
votes
1answer
16 views

Model selection with ncvreg

I'm quite new to R programming and was wondering, How can I read the output of the ncvreg function to find the model it selected (say with the MCP penalty)? Also how can this package be used to ...
0
votes
0answers
5 views

Removing contrast effect in mixed modelling deletion test

I am doing a mixed, linear regression analysis and now want to conduct a series of deletion tests on my max model, where I test if there is a significant increase in deviance once an effect that binds ...
0
votes
0answers
18 views

How to do regression model selection if dummy variables are involved?

Original post on stackoverflow: http://stackoverflow.com/questions/28773153/how-to-do-regression-model-selection-if-dummy-variables-are-involved I am trying to do a logistic regression analysis in R ...
0
votes
0answers
7 views

Text classification: Dealing with potentially false classified instances

My goal is to complete a supervised text classification task with R. There are several classes, of which some of the class counts will be relevant for a subsequent analysis. I have already tried a few ...
1
vote
0answers
24 views

Device Comparision: Correlated or uncorrelated measurements

Background: I want to compare two devices measuring a certain characteristic on a subject. Thereto, each subject is measured once with device A and once with device B. It needs to be assumed that ...
0
votes
0answers
12 views

How to apply AIC to a situation where the mean of a multivariate normal is a 0-1 d-dimensional vector with exactly k 1's

I am trying to apply AIC to estimate mean in the following case: Let us consider that I have $n$ random variables $X_1, \ldots, X_n$, drawn i.i.d. from a normal distribution of mean $\mu\in\{0,1\}^d$ ...
0
votes
0answers
30 views

Model Stacking algorithm

I'm trying the stacking method to see if it improves my results, but before using some R package, i decided to code it by myself. Here's a pseudocode of what i'm doing: ...
1
vote
1answer
35 views

Model selection: can I compare the AIC from models of count data between linear and poisson models?

I am modeling count data (with offset / exposure parameter). My modeling strategy is use of a Poisson model and a negative binomial regression model. I compare model AICs, which are about -760 for my ...
0
votes
0answers
48 views

AIC versus cross validation in time series

I am interested in model selection in a time series setting. For concreteness, suppose I want to select an ARMA model from a pool of ARMA models with different lag orders. The ultimate intent is ...
0
votes
0answers
23 views

Heckman Probit Model the number of explanatory variable in selection model?

I run a Heckman Probit model which is sometimes called as Heckit. It consists two parts like this: |1| Y X1 X2 X3, |2| select(Y2 X1 X2 X3) Y covers Y2 but not vice versa. The question is whether i ...
7
votes
2answers
194 views

Is using correlation matrix to select predictors for regression correct?

A few days ago, a psychologist-researcher of mine told me about his method to select variables to linear regression model. I guess it's not good, but I need to ask someone else to make sure. The ...
1
vote
0answers
40 views

Strictly positive response in regression: what should my “default” model be?

For unbounded continuous responses, Gaussian errors are the analyst's default model for many reasons, one of them being that their ML estimate coincides with the OLS estimate that has many desirable ...
1
vote
1answer
30 views

Proper variable selection: Use only training data or full data?

I'm going through the lab exercises in "Introduction to Statistical Learning" and am having difficulty understanding the proper way to do best subset selection. The book is available here ...
0
votes
0answers
13 views

VAR model selection for forecasting one variable

Suppose I have a VAR model for variables $x_1$ through $x_K$. I will use the model to forecast $x_1$ a few steps ahead and will do this iteratively rather than directly. I am not interested in ...
1
vote
0answers
16 views

Are wrong standard errors a problem if using information theoretic model selection?

In linear regression, if the assumptions of normally distributed residuals and homogenous residuals are broken, incorrect standard errors can be calculated. This can lead to some predictors appearing ...
0
votes
0answers
18 views

Model choice for discrete data with R

I'm keen to model the effect (impact) of variable i.e. var1 and its effect (causation inference?) on var2 given the score measure. Sample data (variation of data I have got) ...
0
votes
0answers
9 views

ensemble model for SVM

I did a nested 5-cv and the resulting models are unstable (high variance among the hyper parameters C and gamma of SVM). So, I don't know how to choose C and gamma for the "final" model. I read that, ...
1
vote
1answer
37 views

Goodness of fit between actual values and non-linear model

I was wanting to get a goodness of fit similar to R^2 for a model I'm evaluating. The output of the model is one of 8 numbers based on environmental characteristics. This is not a linear model, so ...
1
vote
0answers
44 views

GLMM with 2 insignificant variables has lower AIC or BIC compared to same model without those variables…?

I am having a hard time understanding what's going on in with my model selection, and why a model with two insignificant variables is getting chosen as the "best model" over a model without those two ...
2
votes
0answers
52 views

Model selection

I have a small dataset of 37 observations with students' performance on both cognitive tests (5) and professional tests (6). My goal is to predict professional tests (DV) with cognitive tests(IV). To ...
1
vote
0answers
28 views

How to deal with many NAs in dataset when comparing several models?

My task is to compare different logistic regression models to examine which theory better explains the data we have collected. In doing so, I have a set of IVs with a considerable amount of missing ...
0
votes
0answers
21 views

Model Selection Problem

I am asking if there already exist approaches and researches on the following topic. Imagine there are 10 stores and in 3 stores labeled training data was available, so I built 3 classification ...
0
votes
1answer
41 views

Comparison between a multilevel and an unpooled model

Suppose we have fitted two models: a multilevel model and an unpooled model: m1=lmer(y~x+(1|group)) m2=lm(y~x+factor(group)-1) How can I understand which ...
0
votes
1answer
47 views

How to identify the ARCH and GARCH lag lenght in DCC GARCH Model

I just follow the Stata manual for DCC GARCH model. This model contains ARCH(1) and GARCH(1) terms. But my question is, on what basis and how we can can select ...
0
votes
1answer
68 views

Leave-One-Subject-Out cv method

I would like to use a Leave-One-Subject-Out cv on my datasets (I have dataset including 38, 15, 10 participants, respectively). I don't know the hyperparamenters C and gamma of my SVM so I have to ...
6
votes
1answer
93 views

Statistics for model selection and model evaluation

In his Dynamic Econometrics, David F. Hendry issues the following advice: When a 'test' statistic is cited, you must ask 'Was that a selection criterion statistic or a genuine test statistic?' ...
2
votes
1answer
121 views

Cross-validation and logistic regression

I'm interested in building a set of candidate models in R for an analysis using logistic regression. Once I build the set of candidate models and evaluate their fit to the data using AICc (...
2
votes
1answer
57 views

How does step function selects best linear Models which includes polynomial effects and interaction effects in R?

I try to find "best" linear models with continuous and categorical covariables with Interaction Effect by BIC. The continuous covariables should have a quadratic effect on the response variable. ...
3
votes
2answers
73 views

Generate fake data consistent with adjusted R^2 pattern

Is it possible to specify a vector of adjusted $R^2$ values (or any other measure like AIC, BIC, $C_p$) for the set of all possible models in a data set, and then generate data that is consistent with ...
1
vote
1answer
36 views

Is there a default parameter choice for the spike-and-slab prior?

In the spike-and-slab prior, one needs to specify $h_{0j} = P(\beta_j=0)$, which demonstrates our prior belief about how likely $\beta_j$ to be an important predictor. Is there a default choice for ...
0
votes
1answer
29 views

Holdout set for image task

I need to validate whether one or two templates/shapes are present in an image. Fitting two templates has a better maximum likelihood then fitting one template which is a clear symptom of overfitting. ...
0
votes
1answer
85 views

How to use LASSO to select glm model gaussian

I have a small sample size n<20. I want to find which combination of 8 variables better predict y. I was using a stepAICc but it is suggested to away stepwise model selection. I have tried lars ...
0
votes
1answer
48 views

is there a way to plot best glm model in model selection

I have run this glm model y~poly(xa,2)+poly(xb,2)+... Then have found the best fitting model using AICc. The best fitting model has a subset of the ...
0
votes
0answers
33 views

Model selection in nonlinear fitting

Learning curves are fitted with multiple trendlines (exponential, power, logarithm). The fitting is performed by the Levenberg–Marquardt algorithm. So far so good. The question is, how to select the ...
2
votes
2answers
135 views

Using LASSO for variable selection, then using Logit

I know this would muddy the statistical inference, but I am really only concerned with getting as close to an accurate model as I can. I have a dichotomous outcome variable, with a large set of ...
2
votes
1answer
45 views

How is the Akaike Information Criteron applied for model with large number of predictors?

I am reading a paper (details not very relevant) which assumes that the market cost $C$ of a trade is related to $N$ predictors $X_1,\dots,X_N$ (page 25) through a linear relationship $$C = \beta_0 + ...
23
votes
3answers
2k views

Is p-value essentially useless and dangerous to use?

First are some background information. This article "The Odds, Continually Updated" from NY Times happened to catch my attention. To be short, it states that [Bayesian statistics] is proving ...
0
votes
0answers
12 views