# Tagged Questions

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

17 views

### Interaction between 2 quadratic variables

If I have a regression y ~ x1^2 + x1 + x2^2 + x2 + bias, and I want to include interaction between the two quadratic variables, do I make the new regression y ~ x1^2 + x1 + x2^2 + x2 + x1^2*x2^2 + x1^...
20 views

### What tests does the anova() function for model slection use to derive the p-value

I am performing model selection using the anova() function in R. Model 1 is a 'baseline' model involving an interaction of the experimental manipulandum as the predictors. Each subsequent model ...
28 views

### Variability in LASSO models for predicting rare events

I want to build a model for predicting a rare (ca 10%) event in my dataset of around 300 samples and 15 candidate predictors (of these, I know that five, when looked at individually in the whole ...
18 views

### AIC for 2-dimensional data

Let's consider a univariate (experimental) distribution, and two 1-dimensional models to describe it (e.g., a Gaussian distribution and a mixture of two Gaussians). One then computes for each model ...
26 views

### Bayesian Information Criterion(BIC) for gaussian Mixture Models

Say that i have two classes, A and B and some training data $(x_i, y_i), y_i \in {A, B}$. My goal is to fit a mixture model into each class individually and calculate the BIC for the model. The BIC ...
13 views

### Selecting polynomial terms in regression

I'm developing a nonlinear response correction for a sensor (to transform "raw.peak" to "target"). I don't care about interpretability. I do care about future accuracy. One might first just throw it ...
13 views

### Bayesian Entropy Criterion

I am looking at the following paper which introduced the Bayesian Entropy Criterion as a way to select models that minimize the classification error rate: https://hal.inria.fr/inria-00070612/document ...
3 views

### RFE (Recursive Feature Elimination) for Poisson Regression with offset [migrated]

It's my first post so I hope I don't make any editing mistakes. Here's my issue : I'm working on count data and am implementing a Poisson Regression with an exposure factor (that needs to go in the "...
786 views

### Cross-validation misuse (reporting performance for the best hyperparameter value)

Recently I have come across a paper that proposes using a k-NN classifier on an specific dataset. The authors used all the data samples available to perform k-fold cross validation for different k ...
19 views

### Calculate AIC for random model (0 parameters)

I want to compare some action-selection models (e.g. soft-max and epsilon-greedy) to the simplest model I can think of. A random model, one that picks an action randomly among the available ones. To ...
16 views

### Can I compare lme models with and without quadratic (polynomial) term?

I am running mixed-effects analysis using lme package in R. I am trying to understand whether a linear model or a polynomial curve would better capture the change in a variable over time. I know ...
668 views

### A more definitive discussion of variable selection

Background I'm doing clinical research in medicine and have taken several statistics courses. I've never published a paper using linear/logistic regression and would like to do variable selection ...
33 views

### How should one select amongst competing mixed models in a model selection paradigm?

I suspect two biological functions trade off (i.e. as one goes up, the other goes down). Trade-offs are often non-linear, but there has not been a ton of work to suggest which family of curves tends ...
30 views

### Ranking a set of classifiers based on metrics with differing units

Background I'm working on a system that trains an arbitrarily large amount of classifiers (e.g. Support Vector Machines, k-Neighbors Classifiers, Neural Networks, Decision Trees, ...) on the same ...
40 views

### Logistic regression variable selection methods

I'm having trouble to understand Backward elimination in Logistic Regression model. I was looking at this example of Agresti, Categorical Data Analysis, to see how Backward elimination works. What ...
13k views

### How to know that your machine learning problem is hopeless?

Imagine a standard machine-learning scenario: You are confronted with a large multivariate dataset and you have a pretty blurry understanding of it. What you need to do is to make predictions ...
13 views

### How can I compare model fit of LASSO versus OLS?

I want to compare the accuracy of a linear model which uses three predictors and which I estimate with OLS with a model which uses alternative predictors and which I estimate with LASSO. Number of ...
33 views

### Feature Selection with Categorical Variables: Multicollinearity and Statistical Significance

Building a logistic regression model with three categorical features and one continuous. For simplicity, let's say I have the following features and variables: ...
151 views

### Why does -2log-likelihood decrease when you add new variables?

Suppose I'm conducting a study with follow-up survey after N years. Now I want to determine potential risk factors for mortality. I use a model selection procedure and get a table in which the ...
20 views

### Which model is most appropriate to represent this sample data?

I need to determine the most appropriate model to represent the data. It should not be determined from only the $R^2$ value but also from the shape, and need to explain why. What makes one graph ...
23 views

### How to select automatically the best GLMM?

I have a set of 14 variables and I want to construct GLMM's. I want to include at first each variable and then add all the others, one at the time. This will require a lot of combinations of ...
14 views

### Difference between non significant coefficient p-values and variable exclusion using AIC

I am trying to fit a linear regression model with two continuous explanatory variables and one factor with two levels. Rather than predictability, I am especially interested in the interpretability of ...
64 views

### Reducing variable candidates for multivariate regression step by step

I have a set of possible candidates that I want to use in a multivariate regression. I am trying to reduce this set by the following procedure (using Stata): Step 1: univariate regression (if ...
18 views

### Compare predictive power of 1 model on different data

I would like to test if the odds of bets on football matches with higher betting volume are more efficient (i.e. predict the result better) than bets with lower betting volume. I use a probit model ...
39 views

### What selection criteria to use and why? (AIC, RMSE, MAPE) - All possible model selection for time series forecasting

I'm performing all possible model selection in SAS for time series forecasting and basically it is fitting like 40 models on the data and shortlisting the 'n' best models based on a selection criteria....
60 views

### What is wrong with “PROC fishing” syndrome?

RE: https://www.quora.com/What-do-statisticians-e-g-Stats-PhDs-think-of-data-scientists-in-industry-without-stats-backgrounds There are several comments made regarding "PROC FISH syndrome", whereby ...
11 views

### choosing test set in a seasonal time series

The data is sequential, but not necessarily continuous, ie. there are multiple gaps between the start and end date. I fit a regression model, which may or may not involve lagged variables, and I want ...
7 views

### What's the point of 'predictive matching criteria' in Bayesian Statistics?

In this paper, the authors refer to a 'predictive matching criteria'. I fail to see the link between what they write at the beginning of paragraph in page 5(the wrong scale/factor increasing with the ...
26 views

### Model comparison via AIC or BIC for different likelihood maximization procedures

Maximum likelihood estimation of different models (which all model the same variable and assume the same likelihood function) is done by a different method for each model. Simple numerical ...
15 views

### Heckman selection model assumptions

Heckman selection model assumes that 1) error of both selection and main equation are correlated and distributed normally, 2) explanatory variables in selection equation are independent of the error ...
33 views

### Comparing confusion matrices from model fitting

Using R, I got a bunch of confusion matrices from some model fitting. I'm trying to choose the best model by looking at their confusion matrix. Not an easy task. My current method of comparison is to ...
8 views

### Sort or cluster experimental 2D data crossplots with respect to common shape/model

I have a quantity of tabular data, let us say $E=200$ experiments, and $V=20$ variables (all positives). I am trying to find EDA-like dependencies or "correlations" between some of the variables, at ...
29 views

### An interesting two model comparison metric

My colleagues at work use an interesting metric to compare two predictive models that I have never seen before. It can be used for both classification and regression. But we consider classification ...
9 views

### Goodness of fit (R-squared) for SUR

I would like to compare the goodness of fits statistics, $R^2$, of different statistical models estimated by the seemingly unrelated regressions model. In the lecture note of Professor Thorton on SUR,...
25 views

### How are “MuMIn::mod.sel()” and “car::Anova()”/"lmerTest::anova() different for selecting the most optimal model? Is it only the AIC and AICc?

I want to compare several models built using the codes I have written in R for a mixed-effects model. I already knew that anova() function in car package provides <...
46 views

### False discovery rate for across independent models with the same explanatory variables

I have a dataset of >10000 gene expression profiles and I would like to test the effect of 3 explanatory variables and their interactions on each gene expression profile using information criterion (...
46 views

### For model selection/comparison, what kind of test should I use?

I trained and tested two models on the same dataset in a 10-fold cross validation manner. I want to show that one model is supreme than the other. Therefore, I want to show the better model has a ...
11 views

### Does it make sense to compare DIC of model with and without transformed response?

I have a dataset for which I have made two types of longitudinal models. Each of the models has a random intercept and random slope. If the first model has a response Y and covariate X (the random ...
11 views

### Are models within 2 deltaDIC of each other considered equivalent?

I think its a rule when using AIC that should the best model be within 2 unit AIC of the second best model, both are considered with equal weight. Does the same rule-of-thumb apply to Deviance ...
24 views

### include random effect in model

I am having problem deciding wheter to define a variable as a random effect to include in our logistic regression model. Any help on this subject would be most appreciated. In this model, we are ...
99 views

### What does the Akaike Information Criterion (AIC) score of a model mean?

I have seen some questions here about what it means in layman terms, but these are too layman for for my purpose here. I am trying to mathematically understand what does the AIC score mean. But at ...
63 views

### Using DIC to test which data should be used - different sample sizes

I have a glmm model $y \sim b_1 * b_2 * b_3 + random$ where $b_i$ are the fixed effects. I am using DIC to compare models and select the best fitting model. I also have some options in setting up ...
33 views

### Using DIC for model selection: (in)valid comparisons

Imagine I have the following 7 glmm models where $b_1$ through $b_3$ are fixed effects. $M_1 = y \sim b_1 \times b_2 \times b_3$ $M_2 = y \sim b_2 \times b_3$ $M_3 = y \sim b_1 \times b_3$ \$M_4 = ...
1k views

### Variable selection for predictive modeling really needed in 2016?

This question has been asked on CV some yrs ago, it seems worth a repost in light of 1) order of magnitude better computing technology (e.g. parallel computing, HPC etc) and 2) newer techniques, e.g. [...
9 views

### How to select weights in the F-Measure to be aligned with that used in cost-sensitive SVM training?

I am dealing with a classification problem in which Recall is more important than Precision, and the training dataset is an imbalanced one. The approach I am taking is to use oversampling to mitigate ...
53 views

### Model selection in binary logistic regression

I have 3 possible "final" models in binary logistic regression (N=176, Number of events = 36). Now I am trying to decide which one to select. It´s clear,"All models are wrong, but some are useful", ...
85 views

### AICc is picking overly complex models - something stricter?

I'd like to know if there are stricter alternatives to automated model selection than AICc / AIC / BIC. We have approximately ten thousand curves, and for each we'd like to find the most parsimonious ...
23 views

### choosing between traditional statistical models and neural networks using RMSE

I am new to neural networks, and am more familiar with classical linear regression type models. I have a basic question about choosing between the two in attempting to develop a predictive model. Is ...