Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

0
votes
0answers
14 views

Compare predictive power of 1 model on different data

I would like to test if the odds of bets on football matches with higher betting volume are more efficient (i.e. predict the result better) than bets with lower betting volume. I use a probit model ...
1
vote
1answer
18 views

What selection criteria to use and why? (AIC, RMSE, MAPE) - All possible model selection for time series forecasting

I'm performing all possible model selection in SAS for time series forecasting and basically it is fitting like 40 models on the data and shortlisting the 'n' best models based on a selection criteria....
1
vote
2answers
55 views

What is wrong with “PROC fishing” syndrome?

RE: https://www.quora.com/What-do-statisticians-e-g-Stats-PhDs-think-of-data-scientists-in-industry-without-stats-backgrounds There are several comments made regarding "PROC FISH syndrome", whereby ...
0
votes
0answers
11 views

choosing test set in a seasonal time series

The data is sequential, but not necessarily continuous, ie. there are multiple gaps between the start and end date. I fit a regression model, which may or may not involve lagged variables, and I want ...
0
votes
0answers
7 views

What's the point of 'predictive matching criteria' in Bayesian Statistics?

In this paper, the authors refer to a 'predictive matching criteria'. I fail to see the link between what they write at the beginning of paragraph in page 5(the wrong scale/factor increasing with the ...
1
vote
0answers
25 views

Model comparison via AIC or BIC for different likelihood maximization procedures

Maximum likelihood estimation of different models (which all model the same variable and assume the same likelihood function) is done by a different method for each model. Simple numerical ...
1
vote
0answers
12 views

Heckman selection model assumptions

Heckman selection model assumes that 1) error of both selection and main equation are correlated and distributed normally, 2) explanatory variables in selection equation are independent of the error ...
1
vote
1answer
28 views

Comparing confusion matrices from model fitting

Using R, I got a bunch of confusion matrices from some model fitting. I'm trying to choose the best model by looking at their confusion matrix. Not an easy task. My current method of comparison is to ...
0
votes
0answers
6 views

Sort or cluster experimental 2D data crossplots with respect to common shape/model

I have a quantity of tabular data, let us say $E=200$ experiments, and $V=20$ variables (all positives). I am trying to find EDA-like dependencies or "correlations" between some of the variables, at ...
0
votes
0answers
26 views

An interesting two model comparison metric

My colleagues at work use an interesting metric to compare two predictive models that I have never seen before. It can be used for both classification and regression. But we consider classification ...
0
votes
0answers
7 views

Goodness of fit (R-squared) for SUR

I would like to compare the goodness of fits statistics, $R^2$, of different statistical models estimated by the seemingly unrelated regressions model. In the lecture note of Professor Thorton on SUR,...
0
votes
0answers
19 views

How are “MuMIn::mod.sel()” and “car::Anova()”/"lmerTest::anova() different for selecting the most optimal model? Is it only the AIC and AICc?

I want to compare several models built using the codes I have written in R for a mixed-effects model. I already knew that anova() function in car package provides <...
1
vote
0answers
43 views

False discovery rate for across independent models with the same explanatory variables

I have a dataset of >10000 gene expression profiles and I would like to test the effect of 3 explanatory variables and their interactions on each gene expression profile using information criterion (...
1
vote
1answer
43 views

For model selection/comparison, what kind of test should I use?

I trained and tested two models on the same dataset in a 10-fold cross validation manner. I want to show that one model is supreme than the other. Therefore, I want to show the better model has a ...
0
votes
0answers
11 views

Does it make sense to compare DIC of model with and without transformed response?

I have a dataset for which I have made two types of longitudinal models. Each of the models has a random intercept and random slope. If the first model has a response Y and covariate X (the random ...
0
votes
0answers
11 views

Are models within 2 deltaDIC of each other considered equivalent?

I think its a rule when using AIC that should the best model be within 2 unit AIC of the second best model, both are considered with equal weight. Does the same rule-of-thumb apply to Deviance ...
0
votes
0answers
4 views

Find the selected variables for regsubset in R [migrated]

When using forward or backward selection in R, for example, backwardselect <- regsubsets(x = df[,-c(1,2)],y = df[,1], nvma = 10, method="backward") backwardselect <- summary(backwardselect) ...
0
votes
0answers
23 views

include random effect in model

I am having problem deciding wheter to define a variable as a random effect to include in our logistic regression model. Any help on this subject would be most appreciated. In this model, we are ...
7
votes
0answers
49 views

What does the Akaike Information Criterion (AIC) score of a model mean?

I have seen some questions here about what it means in layman terms, but these are too layman for for my purpose here. I am trying to mathematically understand what does the AIC score mean. But at ...
1
vote
1answer
43 views
+50

Using DIC to test which data should be used - different sample sizes

I have a glmm model $y \sim b_1 * b_2 * b_3 + random$ where $b_i$ are the fixed effects. I am using DIC to compare models and select the best fitting model. I also have some options in setting up ...
1
vote
1answer
29 views

Using DIC for model selection: (in)valid comparisons

Imagine I have the following 7 glmm models where $b_1$ through $b_3$ are fixed effects. $M_1 = y \sim b_1 \times b_2 \times b_3$ $M_2 = y \sim b_2 \times b_3$ $M_3 = y \sim b_1 \times b_3$ $M_4 = ...
31
votes
6answers
1k views

Variable selection for predictive modeling really needed in 2016?

This question has been asked on CV some yrs ago, it seems worth a repost in light of 1) order of magnitude better computing technology (e.g. parallel computing, HPC etc) and 2) newer techniques, e.g. [...
0
votes
0answers
8 views

How to select weights in the F-Measure to be aligned with that used in cost-sensitive SVM training?

I am dealing with a classification problem in which Recall is more important than Precision, and the training dataset is an imbalanced one. The approach I am taking is to use oversampling to mitigate ...
1
vote
0answers
51 views

Model selection in binary logistic regression

I have 3 possible "final" models in binary logistic regression (N=176, Number of events = 36). Now I am trying to decide which one to select. It´s clear,"All models are wrong, but some are useful", ...
3
votes
2answers
80 views

AICc is picking overly complex models - something stricter?

I'd like to know if there are stricter alternatives to automated model selection than AICc / AIC / BIC. We have approximately ten thousand curves, and for each we'd like to find the most parsimonious ...
1
vote
1answer
21 views

choosing between traditional statistical models and neural networks using RMSE

I am new to neural networks, and am more familiar with classical linear regression type models. I have a basic question about choosing between the two in attempting to develop a predictive model. Is ...
1
vote
1answer
37 views

General-to-specific subset selection (“Autometrics”) performing well in macroeconomics

I wonder why general-to-specific (GETS) subset selection and particularly the Autometrics algorithm are performing well in macroeconomic modelling/forecasting. How does Autometrics work? Doornik "...
0
votes
0answers
35 views

Multiple testing & model selection using AICc

I have a situation very similar to this post. I am finding the best-fit mixed effects model among a set of 5 candidate models for 7 different dependent variables (y1 to y7) using the same dataset of ...
0
votes
0answers
36 views

assessing confidence of simple likelihood ratio test

I have a Gaussian mixture model of two normal distributions, which are scaled relatively to each other by a factor $\alpha$. I observe the data vector $x$ with values from $x_1$ to $x_n$, which is ...
3
votes
3answers
231 views

Does likelihood ratio test control for overfitting?

I have two nested logistic regression models, A and B. A is nested under B. Let's say B has $K$ more features than A. B has a higher log likelihood than A. However the improved likelihood of B is due ...
0
votes
0answers
16 views

Normal vs. non normal distributed data. Choice of model

I have a question about normally distributed and non normally distributed data. I am going to do a genotype association analysis but I want to build a model for each trait that I have to know which ...
1
vote
0answers
23 views

Why is it recommended to delete interaction terms “as a chunk”?

In this answer it was discussed that when terms are being deleted for the purposes of model simplification, it's best to delete terms "by chunk". That is, in an x1 * x2 * x3 * x4 scenario we'd ...
0
votes
1answer
28 views

K-fold CV based model selection with a constraint on the number of features?

I am currently working on project where I need to train a logistic regression classifier with a combined $l_1$/$l_2$-penalty that satisfies a hard on the number of features. Specifically, my dataset ...
1
vote
1answer
27 views

Model selection in PGLS?

I am using Phylogenetic General Least Squares in the R package 'caper'. I have 4 predictor variables and I would like to know which are correlated with my response variable, while taking the ...
0
votes
1answer
39 views

ARMA model selection: in-sample vs. out-sample accuracy measures

I have a time series for 1000 days for many firms. I am interested to know, in general, on what basis I should select an ARMA model (the nature of my problem restricts integration order to 0). Should ...
0
votes
1answer
26 views

ARIMA Time series forecasting in R, help on choosing adequate model

I am trying to create adequate time series model in R. I have doubt about adequacy. My data is year and total number of events: ...
0
votes
0answers
18 views

Confusion over the test error and the expected error and actual implementation

Given a training set, $\tau = \{(x_1,y_1,\dots, x_N,y_N \}$ and a model $\hat{f}(x)$ has been fit. We have the following two definitions: The Generalisation (Test) Error $$ Err(\tau) = E(L(Y,\hat{f}(...
0
votes
1answer
51 views

AIC, model selection and overfitting

I am looking for references that specifically show that Akaike's Information Criterion (AIC), or its corrected form (AICc), can in some practical applications -- that is, not in the asymptotic regime -...
3
votes
1answer
35 views

Is AIC a valid criterion for the selection of variance structures in GLS?

In generalized least squares, I’ve specified a weights function that accounts for heterogeneity in residual variance that exist along the range of a covariate. The validation graphs (residuals ...
1
vote
1answer
46 views

What comes first: outlier detection or model selection?

I'm fitting a GLMM (mixed logistic regression) in R. I have five covariates. For model selection, I'm using glmmLasso() (in R) to determine which of the five covariates and their interactions should ...
1
vote
0answers
14 views

Should I run variable selection within MICE for Multiple Imputation?

I have a dataset with around 100 variables, and I plan on getting multiply imputed datasets using the mice package in R. The literature I have read seems to advocate regressing each variable against ...
3
votes
0answers
16 views

PDF/CDF ratio interpretation

What would be the intuitive interpretation of the ratio of probability and cummulative density function during the first stage of Heckman selection model? Is it the likelihood of selection for the ...
0
votes
0answers
3 views

What component of the result should I look at when doing a LLM model fit?

I am running mixed effects models with poisson and negative binomial fits. To asses which of the models are better, what components of the models should I look at? Some popular methods I follow: a) ...
0
votes
0answers
16 views

range of possible values and model complexity

I have a prediction model where the likelihood is a function of two components $(A,B)$ $L \sim \bf{A}^\alpha + \bf{B}^\beta$ , where both $\bf{A}$ and $\bf{B}$ are $n\times n$ matrices ($n$ is the ...
0
votes
0answers
14 views

Cancelling roots in ARMA(1,1) with external regressors

I am trying to find out what cancelling roots would imply for the estimators of my external regressors in my ARMA(1,1) model. Unfortunately however I'm stuck in my final step since I'm insecure about ...
0
votes
0answers
34 views

Help discussing stationarity using correlograms? ARMA/ARIMA modelling

I am currently trying to understand how to use correlograms to examine stationarity and analysis the appropriate models. Please can you advice, below I have included my ACFs and PACFs, and I am trying ...
1
vote
0answers
19 views

Likelihood ratio test seems to show little difference between models with AICc difference of 3

I'm running a multinomial logistic regression analysis of the behavioural responses of deer to camera traps using no reaction, reaction and strong reaction as dependent variables and season, camera ...
0
votes
1answer
42 views

Using Information theory with all possible models to select the best Model

I have a simple data set to find out about the effect of cultivation period length on soil organisms. The main factor of interest is age_class, a categorical variable defining the age of a field under ...
3
votes
0answers
49 views

AIC equivalent to Mallows' Cp and Mallows' Cp unbiased for test MSE

Part 1: The goal is to show that with Gaussian errors and a linear model, Mallows' $C_p$ and $AIC$ are equivalent. Using our definition of Mallows' $C_p$: $$C_p=1/n(RSS+2d\hat\sigma^2)$$ and $AIC$: $$...
1
vote
1answer
35 views

Comparing the distribution fits of a bivariate and a univariate model

Suppose I've done an experiment and I have a distribution of observations $x$ that vary between $-\pi$ and $\pi$. Now suppose each $x$ is associated with a second observation $y$ that may or may not ...