Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

Filter by
Sorted by
Tagged with
224
votes
8answers
107k views

Algorithms for automatic model selection

I would like to implement an algorithm for automatic model selection. I am thinking of doing stepwise regression but anything will do (it has to be based on linear regressions though). My problem ...
249
votes
3answers
24k views

How to know that your machine learning problem is hopeless?

Imagine a standard machine-learning scenario: You are confronted with a large multivariate dataset and you have a pretty blurry understanding of it. What you need to do is to make predictions ...
127
votes
4answers
61k views

Nested cross validation for model selection

How can one use nested cross validation for model selection? From what I read online, nested CV works as follows: There is the inner CV loop, where we may conduct a grid search (e.g. running K-fold ...
171
votes
5answers
59k views

Training on the full dataset after cross-validation?

TL:DR: Is it ever a good idea to train an ML model on all the data available before shipping it to production? Put another way, is it ever ok to train on all data available and not check if the model ...
260
votes
12answers
192k views

Is there any reason to prefer the AIC or BIC over the other?

The AIC and BIC are both methods of assessing model fit penalized for the number of estimated parameters. As I understand it, BIC penalizes models more for free parameters than does AIC. Beyond a ...
230
votes
7answers
162k views

How to choose a predictive model after k-fold cross-validation?

I am wondering how to choose a predictive model after doing K-fold cross-validation. This may be awkwardly phrased, so let me explain in more detail: whenever I run K-fold cross-validation, I use K ...
17
votes
5answers
13k views

Can I ignore coefficients for non-significant levels of factors in a linear model?

After seeking clarification about linear model coefficients over here I have a follow up question concerning non-signficant (high p value) for coefficients of factor levels. Example: If my linear ...
722
votes
11answers
865k views

How to choose the number of hidden layers and nodes in a feedforward neural network?

Is there a standard and accepted method for selecting the number of layers, and the number of nodes in each layer, in a feed-forward neural network? I'm interested in automated ways of building neural ...
86
votes
5answers
22k views

What are modern, easily used alternatives to stepwise regression?

I have a dataset with around 30 independent variables and would like to construct a generalized linear model (GLM) to explore the relationship between them and the dependent variable. I am aware that ...
28
votes
3answers
19k views

Prerequisites for AIC model comparison

What are exactly the prerequisites, that need to be fulfilled for AIC model comparison to work? I just came around this question when I did comparison like this: ...
42
votes
1answer
19k views

What are posterior predictive checks and what makes them useful?

I understand what the posterior predictive distribution is, and I have been reading about posterior predictive checks, although it isn't clear to me what it does yet. What exactly is the posterior ...
95
votes
2answers
13k views

How much do we know about p-hacking "in the wild"?

The phrase p-hacking (also: "data dredging", "snooping" or "fishing") refers to various kinds of statistical malpractice in which results become artificially statistically significant. There are many ...
59
votes
3answers
42k views

Linear model with log-transformed response vs. generalized linear model with log link

In this paper titled "CHOOSING AMONG GENERALIZED LINEAR MODELS APPLIED TO MEDICAL DATA" the authors write: In a generalized linear model, the mean is transformed, by the link function, instead of ...
77
votes
6answers
11k views

Variable selection for predictive modeling really needed in 2016?

This question has been asked on CV some yrs ago, it seems worth a repost in light of 1) order of magnitude better computing technology (e.g. parallel computing, HPC etc) and 2) newer techniques, e.g. [...
43
votes
3answers
19k views

whether to rescale indicator / binary / dummy predictors for LASSO

For the LASSO (and other model selecting procedures) it is crucial to rescale the predictors. The general recommendation I follow is simply to use a 0 mean, 1 standard deviation normalization for ...
25
votes
3answers
5k views

Paradox in model selection (AIC, BIC, to explain or to predict?)

Having read Galit Shmueli's "To Explain or to Predict" (2010) and some literature on model selection using AIC and BIC, I am puzzled by an apparent contradiction. There are three premises, AIC- ...
31
votes
3answers
5k views

How to build the final model and tune probability threshold after nested cross-validation?

Firstly, apologies for posting a question that has already been discussed at length here, here, here, here, here, and for reheating an old topic. I know @DikranMarsupial has written about this topic ...
13
votes
2answers
7k views

What are chunk tests?

In answer to a question on model selection in the presence of multicollinearity, Frank Harrell suggested: Put all variables in the model but do not test for the effect of one variable adjusted for ...
7
votes
1answer
3k views

If a factor variable is to be dropped in model selection, should all levels be dropped simultaneously? If so, why? [duplicate]

In answer to a previous question factor pooling in model selection was discussed. If a factor or categorical variable is to be dropped in model selection, should all levels be dropped simultaneously? ...
64
votes
2answers
35k views

Why only three partitions? (training, validation, test)

When you are trying to fit models to a large dataset, the common advice is to partition the data into three parts: the training, validation, and test dataset. This is because the models usually have ...
37
votes
2answers
12k views

Model selection and cross-validation: The right way

There are numerous threads in CrossValidated on the topic of model selection and cross validation. Here are a few: Internal vs external cross-validation and model selection @DikranMarsupial's top ...
47
votes
7answers
140k views

Choosing variables to include in a multiple linear regression model

I am currently working to build a model using a multiple linear regression. After fiddling around with my model, I am unsure how to best determine which variables to keep and which to remove. My ...
19
votes
2answers
40k views

Is using correlation matrix to select predictors for regression correct?

A few days ago, a psychologist-researcher of mine told me about his method to select variables to linear regression model. I guess it's not good, but I need to ask someone else to make sure. The ...
4
votes
1answer
11k views

Lasso regression coefficients values

I applied lasso regression for variable selection, and out of 10 variables lasso selected 4 variables. ...
32
votes
4answers
81k views

Choosing the best model from among different "best" models

How do you choose a model from among different models chosen by different methods (e.g. backwards or forwards selection)? Also what is a parsimonious model?
16
votes
2answers
17k views

Understanding bootstrapping for validation and model selection

I think I understand how the fundamentals of bootstrapping work, but I'm not sure I understand how I can use bootstrapping for model selection or to avoid overfitting. For model selection, for ...
27
votes
1answer
4k views

Choosing among proper scoring rules

Most resources on proper scoring rules mention a number of different scoring rules like log-loss, Brier score or spherical scoring. However, they often don't give much guidance on the differences ...
34
votes
3answers
12k views

Can AIC compare across different types of model?

I'm using AIC (Akaike's Information Criterion) to compare non-linear models in R. Is it valid to compare the AICs of different types of model? Specifically, I'm comparing a model fitted by glm versus ...
11
votes
1answer
15k views

AIC versus Likelihood Ratio Test in Model Variable Selection

The software that I am currently using to build a model compares a "current run" model to a "reference model" and reports (where applicable) both a chi-squared p-value based on likelihood ratio tests ...
25
votes
3answers
95k views

AIC or p-value: which one to choose for model selection?

I'm brand new to this R thing but am unsure which model to select. I did a stepwise forward regression selecting each variable based on the lowest AIC. I came up with 3 models that I'm unsure which ...
49
votes
3answers
10k views

Empirical justification for the one standard error rule when using cross-validation

Are there any empirical studies justifying the use of the one standard error rule in favour of parsimony? Obviously it depends on the data-generation process of the data, but anything which analyses a ...
19
votes
4answers
14k views

Determining best fitting curve fitting function out of linear, exponential, and logarithmic functions

Context: From a question on Mathematics Stack Exchange (Can I build a program), someone has a set of $x-y$ points, and wants to fit a curve to it, linear, exponential or logarithmic. The usual ...
17
votes
2answers
4k views

Best suggested textbooks on Bootstrap resampling?

I just wanted to ask which are in your opinion the best available books on bootstrap out there. By this I don't necessarily only mean the one written by its developers. Could you please indicate ...
12
votes
2answers
7k views

How to use principal components analysis to select variables for regression?

I am currently using principal components analysis to select variables to use in modelling. At the moment, I make measurements A, B and C in my experiments -- What I really want to know is: Can I ...
13
votes
1answer
2k views

Can you compare AIC values as long as the models are based on the same dataset?

I am doing some forecasting in R using Rob Hyndman's forecast package. The paper belonging to the package can be found here. In the paper, after explaining the automatic forecasting algorithms, the ...
10
votes
1answer
2k views

Equivalence of AIC and p-values in model selection

In a comment to the answer of this question, it was stated that using AIC in model selection was equivalent to using a p-value of 0.154. I tried it in R, where I used a "backward" subset selection ...
4
votes
2answers
5k views

Model uncertainty (model averaging) and R-Squared ($R^2$)

Is it possible to calculate r-squared for an "average model"? Lets say I have 4 different response variables that I want to model to a set (or subset) of 4 independent variables. I'd then like to ...
62
votes
2answers
3k views

A more definitive discussion of variable selection

Background I'm doing clinical research in medicine and have taken several statistics courses. I've never published a paper using linear/logistic regression and would like to do variable selection ...
44
votes
5answers
203k views

AIC guidelines in model selection

I typically use BIC as my understanding is that it values parsimony more strongly than does AIC. However, I have decided to use a more comprehensive approach now and would like to use AIC as well. I ...
23
votes
2answers
8k views

Cross Validation (error generalization) after model selection

Note: Case is n>>p I am reading Elements of Statistical Learning and there are various mentions about the "right" way to do cross validation( e.g. page 60, page 245). Specifically, my question is how ...
38
votes
6answers
43k views

How does cross-validation overcome the overfitting problem?

Why does a cross-validation procedure overcome the problem of overfitting a model?
22
votes
7answers
13k views

Measures of model complexity

How can we compare complexity of two models with the same number of parameters? Edit 09/19: To clarify, model complexity is a measure of how hard it is to learn from limited data. When two models fit ...
27
votes
3answers
21k views

ROC vs Precision-recall curves on imbalanced dataset

I just finished reading this discussion. They argue that PR AUC is better than ROC AUC on imbalanced dataset. For example, we have 10 samples in test dataset. 9 samples are positive and 1 is ...
28
votes
3answers
4k views

AIC versus cross validation in time series: the small sample case

I am interested in model selection in a time series setting. For concreteness, suppose I want to select an ARMA model from a pool of ARMA models with different lag orders. The ultimate intent is ...
13
votes
3answers
10k views

When is it appropriate to select models by minimising the AIC?

It is well established, at least among statisticians of some higher calibre, that models with the values of the AIC statistic within a certain threshold of the minimum value should be considered as ...
18
votes
2answers
3k views

Are there any circumstances where stepwise regression should be used?

Stepwise regression had been overused in many biomedical papers in the past but this appears to be improving with better education of its many issues. Many older reviewers however do still ask for it. ...
7
votes
1answer
309 views

Effects of model selection and misspecification testing on inference: Probabilistic Reduction approach (Aris Spanos)

This question is about pre-test bias, inference after model selection and data snooping within the Probabilistic Reduction (PR) methodology by Aris Spanos (which is related to the Error Statistics ...
6
votes
1answer
3k views

For model selection/comparison, what kind of test should I use?

I trained and tested two models on the same dataset in a 10-fold cross validation manner. I want to show that one model is supreme than the other. Therefore, I want to show the better model has a ...
5
votes
1answer
2k views

Zero inflate models vs generalized mixture model

I am looking to compare the fit of a zero-inflated mixture model and a Poisson mixture model. The random effects in both models are different. Comparing the fitted values of both models ignores the ...
7
votes
3answers
313 views

Justification for and optimality of $R^2_{adj.}$ as a model selection criterion

In a recent thread, use of adjusted $R^2$ ($R^2_{adj.}$) is mentioned in the context of model selection, e.g. The adjustment was invented as a solution to problems caused by variable selection ...

1
2 3 4 5
7