Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

0
votes
0answers
17 views

Interaction between 2 quadratic variables

If I have a regression y ~ x1^2 + x1 + x2^2 + x2 + bias, and I want to include interaction between the two quadratic variables, do I make the new regression y ~ x1^2 + x1 + x2^2 + x2 + x1^2*x2^2 + x1^...
0
votes
0answers
20 views

What tests does the anova() function for model slection use to derive the p-value

I am performing model selection using the anova() function in R. Model 1 is a 'baseline' model involving an interaction of the experimental manipulandum as the predictors. Each subsequent model ...
4
votes
0answers
28 views

Variability in LASSO models for predicting rare events

I want to build a model for predicting a rare (ca 10%) event in my dataset of around 300 samples and 15 candidate predictors (of these, I know that five, when looked at individually in the whole ...
1
vote
0answers
18 views

AIC for 2-dimensional data

Let's consider a univariate (experimental) distribution, and two 1-dimensional models to describe it (e.g., a Gaussian distribution and a mixture of two Gaussians). One then computes for each model $$...
0
votes
0answers
26 views

Bayesian Information Criterion(BIC) for gaussian Mixture Models

Say that i have two classes, A and B and some training data $(x_i, y_i), y_i \in {A, B}$. My goal is to fit a mixture model into each class individually and calculate the BIC for the model. The BIC ...
1
vote
0answers
13 views

Selecting polynomial terms in regression

I'm developing a nonlinear response correction for a sensor (to transform "raw.peak" to "target"). I don't care about interpretability. I do care about future accuracy. One might first just throw it ...
0
votes
0answers
13 views

Bayesian Entropy Criterion

I am looking at the following paper which introduced the Bayesian Entropy Criterion as a way to select models that minimize the classification error rate: https://hal.inria.fr/inria-00070612/document ...
0
votes
0answers
3 views

RFE (Recursive Feature Elimination) for Poisson Regression with offset [migrated]

It's my first post so I hope I don't make any editing mistakes. Here's my issue : I'm working on count data and am implementing a Poisson Regression with an exposure factor (that needs to go in the "...
24
votes
1answer
786 views

Cross-validation misuse (reporting performance for the best hyperparameter value)

Recently I have come across a paper that proposes using a k-NN classifier on an specific dataset. The authors used all the data samples available to perform k-fold cross validation for different k ...
0
votes
0answers
19 views

Calculate AIC for random model (0 parameters)

I want to compare some action-selection models (e.g. soft-max and epsilon-greedy) to the simplest model I can think of. A random model, one that picks an action randomly among the available ones. To ...
1
vote
1answer
16 views

Can I compare lme models with and without quadratic (polynomial) term?

I am running mixed-effects analysis using lme package in R. I am trying to understand whether a linear model or a polynomial curve would better capture the change in a variable over time. I know ...
28
votes
2answers
668 views

A more definitive discussion of variable selection

Background I'm doing clinical research in medicine and have taken several statistics courses. I've never published a paper using linear/logistic regression and would like to do variable selection ...
0
votes
0answers
33 views

How should one select amongst competing mixed models in a model selection paradigm?

I suspect two biological functions trade off (i.e. as one goes up, the other goes down). Trade-offs are often non-linear, but there has not been a ton of work to suggest which family of curves tends ...
1
vote
1answer
30 views

Ranking a set of classifiers based on metrics with differing units

Background I'm working on a system that trains an arbitrarily large amount of classifiers (e.g. Support Vector Machines, k-Neighbors Classifiers, Neural Networks, Decision Trees, ...) on the same ...
2
votes
1answer
40 views

Logistic regression variable selection methods

I'm having trouble to understand Backward elimination in Logistic Regression model. I was looking at this example of Agresti, Categorical Data Analysis, to see how Backward elimination works. What ...
129
votes
3answers
13k views

How to know that your machine learning problem is hopeless?

Imagine a standard machine-learning scenario: You are confronted with a large multivariate dataset and you have a pretty blurry understanding of it. What you need to do is to make predictions ...
0
votes
0answers
13 views

How can I compare model fit of LASSO versus OLS?

I want to compare the accuracy of a linear model which uses three predictors and which I estimate with OLS with a model which uses alternative predictors and which I estimate with LASSO. Number of ...
2
votes
0answers
33 views

Feature Selection with Categorical Variables: Multicollinearity and Statistical Significance

Building a logistic regression model with three categorical features and one continuous. For simplicity, let's say I have the following features and variables: ...
3
votes
2answers
151 views

Why does -2log-likelihood decrease when you add new variables?

Suppose I'm conducting a study with follow-up survey after N years. Now I want to determine potential risk factors for mortality. I use a model selection procedure and get a table in which the ...
0
votes
1answer
20 views

Which model is most appropriate to represent this sample data?

I need to determine the most appropriate model to represent the data. It should not be determined from only the $R^2$ value but also from the shape, and need to explain why. What makes one graph ...
0
votes
2answers
23 views

How to select automatically the best GLMM?

I have a set of 14 variables and I want to construct GLMM's. I want to include at first each variable and then add all the others, one at the time. This will require a lot of combinations of ...
0
votes
0answers
14 views

Difference between non significant coefficient p-values and variable exclusion using AIC

I am trying to fit a linear regression model with two continuous explanatory variables and one factor with two levels. Rather than predictability, I am especially interested in the interpretability of ...
1
vote
0answers
64 views

Reducing variable candidates for multivariate regression step by step

I have a set of possible candidates that I want to use in a multivariate regression. I am trying to reduce this set by the following procedure (using Stata): Step 1: univariate regression (if ...
0
votes
0answers
18 views

Compare predictive power of 1 model on different data

I would like to test if the odds of bets on football matches with higher betting volume are more efficient (i.e. predict the result better) than bets with lower betting volume. I use a probit model ...
1
vote
1answer
39 views

What selection criteria to use and why? (AIC, RMSE, MAPE) - All possible model selection for time series forecasting

I'm performing all possible model selection in SAS for time series forecasting and basically it is fitting like 40 models on the data and shortlisting the 'n' best models based on a selection criteria....
1
vote
2answers
60 views

What is wrong with “PROC fishing” syndrome?

RE: https://www.quora.com/What-do-statisticians-e-g-Stats-PhDs-think-of-data-scientists-in-industry-without-stats-backgrounds There are several comments made regarding "PROC FISH syndrome", whereby ...
0
votes
0answers
11 views

choosing test set in a seasonal time series

The data is sequential, but not necessarily continuous, ie. there are multiple gaps between the start and end date. I fit a regression model, which may or may not involve lagged variables, and I want ...
0
votes
0answers
7 views

What's the point of 'predictive matching criteria' in Bayesian Statistics?

In this paper, the authors refer to a 'predictive matching criteria'. I fail to see the link between what they write at the beginning of paragraph in page 5(the wrong scale/factor increasing with the ...
1
vote
0answers
26 views

Model comparison via AIC or BIC for different likelihood maximization procedures

Maximum likelihood estimation of different models (which all model the same variable and assume the same likelihood function) is done by a different method for each model. Simple numerical ...
1
vote
0answers
15 views

Heckman selection model assumptions

Heckman selection model assumes that 1) error of both selection and main equation are correlated and distributed normally, 2) explanatory variables in selection equation are independent of the error ...
1
vote
1answer
33 views

Comparing confusion matrices from model fitting

Using R, I got a bunch of confusion matrices from some model fitting. I'm trying to choose the best model by looking at their confusion matrix. Not an easy task. My current method of comparison is to ...
0
votes
0answers
8 views

Sort or cluster experimental 2D data crossplots with respect to common shape/model

I have a quantity of tabular data, let us say $E=200$ experiments, and $V=20$ variables (all positives). I am trying to find EDA-like dependencies or "correlations" between some of the variables, at ...
0
votes
0answers
29 views

An interesting two model comparison metric

My colleagues at work use an interesting metric to compare two predictive models that I have never seen before. It can be used for both classification and regression. But we consider classification ...
0
votes
0answers
9 views

Goodness of fit (R-squared) for SUR

I would like to compare the goodness of fits statistics, $R^2$, of different statistical models estimated by the seemingly unrelated regressions model. In the lecture note of Professor Thorton on SUR,...
0
votes
1answer
25 views

How are “MuMIn::mod.sel()” and “car::Anova()”/"lmerTest::anova() different for selecting the most optimal model? Is it only the AIC and AICc?

I want to compare several models built using the codes I have written in R for a mixed-effects model. I already knew that anova() function in car package provides <...
1
vote
0answers
46 views

False discovery rate for across independent models with the same explanatory variables

I have a dataset of >10000 gene expression profiles and I would like to test the effect of 3 explanatory variables and their interactions on each gene expression profile using information criterion (...
1
vote
1answer
46 views

For model selection/comparison, what kind of test should I use?

I trained and tested two models on the same dataset in a 10-fold cross validation manner. I want to show that one model is supreme than the other. Therefore, I want to show the better model has a ...
0
votes
0answers
11 views

Does it make sense to compare DIC of model with and without transformed response?

I have a dataset for which I have made two types of longitudinal models. Each of the models has a random intercept and random slope. If the first model has a response Y and covariate X (the random ...
0
votes
0answers
11 views

Are models within 2 deltaDIC of each other considered equivalent?

I think its a rule when using AIC that should the best model be within 2 unit AIC of the second best model, both are considered with equal weight. Does the same rule-of-thumb apply to Deviance ...
0
votes
0answers
24 views

include random effect in model

I am having problem deciding wheter to define a variable as a random effect to include in our logistic regression model. Any help on this subject would be most appreciated. In this model, we are ...
12
votes
0answers
99 views

What does the Akaike Information Criterion (AIC) score of a model mean?

I have seen some questions here about what it means in layman terms, but these are too layman for for my purpose here. I am trying to mathematically understand what does the AIC score mean. But at ...
2
votes
1answer
63 views

Using DIC to test which data should be used - different sample sizes

I have a glmm model $y \sim b_1 * b_2 * b_3 + random$ where $b_i$ are the fixed effects. I am using DIC to compare models and select the best fitting model. I also have some options in setting up ...
1
vote
1answer
33 views

Using DIC for model selection: (in)valid comparisons

Imagine I have the following 7 glmm models where $b_1$ through $b_3$ are fixed effects. $M_1 = y \sim b_1 \times b_2 \times b_3$ $M_2 = y \sim b_2 \times b_3$ $M_3 = y \sim b_1 \times b_3$ $M_4 = ...
34
votes
6answers
1k views

Variable selection for predictive modeling really needed in 2016?

This question has been asked on CV some yrs ago, it seems worth a repost in light of 1) order of magnitude better computing technology (e.g. parallel computing, HPC etc) and 2) newer techniques, e.g. [...
0
votes
0answers
9 views

How to select weights in the F-Measure to be aligned with that used in cost-sensitive SVM training?

I am dealing with a classification problem in which Recall is more important than Precision, and the training dataset is an imbalanced one. The approach I am taking is to use oversampling to mitigate ...
1
vote
0answers
53 views

Model selection in binary logistic regression

I have 3 possible "final" models in binary logistic regression (N=176, Number of events = 36). Now I am trying to decide which one to select. It´s clear,"All models are wrong, but some are useful", ...
3
votes
2answers
85 views

AICc is picking overly complex models - something stricter?

I'd like to know if there are stricter alternatives to automated model selection than AICc / AIC / BIC. We have approximately ten thousand curves, and for each we'd like to find the most parsimonious ...
1
vote
1answer
23 views

choosing between traditional statistical models and neural networks using RMSE

I am new to neural networks, and am more familiar with classical linear regression type models. I have a basic question about choosing between the two in attempting to develop a predictive model. Is ...
1
vote
1answer
39 views

General-to-specific subset selection (“Autometrics”) performing well in macroeconomics

I wonder why general-to-specific (GETS) subset selection and particularly the Autometrics algorithm are performing well in macroeconomic modelling/forecasting. How does Autometrics work? Doornik "...
0
votes
0answers
38 views

Multiple testing & model selection using AICc

I have a situation very similar to this post. I am finding the best-fit mixed effects model among a set of 5 candidate models for 7 different dependent variables (y1 to y7) using the same dataset of ...