Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

1
vote
0answers
36 views

Model selection in binary logistic regression

I have 3 possible "final" models in binary logistic regression (N=176, Number of events = 36). Now I am trying to decide which one to select. It´s clear,"All models are wrong, but some are useful", ...
2
votes
2answers
66 views

AICc is picking overly complex models - something stricter?

I'd like to know if there are stricter alternatives to automated model selection than AICc / AIC / BIC. We have approximately ten thousand curves, and for each we'd like to find the most parsimonious ...
1
vote
1answer
20 views

choosing between traditional statistical models and neural networks using RMSE

I am new to neural networks, and am more familiar with classical linear regression type models. I have a basic question about choosing between the two in attempting to develop a predictive model. Is ...
1
vote
1answer
29 views

General-to-specific subset selection (“Autometrics”) performing well in macroeconomics

I wonder why general-to-specific (GETS) subset selection and particularly the Autometrics algorithm are performing well in macroeconomic modelling/forecasting. How does Autometrics work? Doornik ...
0
votes
0answers
27 views

Multiple testing & model selection using AICc

I have a situation very similar to this post. I am finding the best-fit mixed effects model among a set of 5 candidate models for 7 different dependent variables (y1 to y7) using the same dataset of ...
0
votes
0answers
25 views

assessing confidence of simple likelihood ratio test

I have a Gaussian mixture model of two normal distributions, which are scaled relatively to each other by a factor $\alpha$. I observe the data vector $x$ with values from $x_1$ to $x_n$, which is ...
3
votes
3answers
215 views

Does likelihood ratio test control for overfitting?

I have two nested logistic regression models, A and B. A is nested under B. Let's say B has $K$ more features than A. B has a higher log likelihood than A. However the improved likelihood of B is due ...
0
votes
0answers
11 views

GLM and GLMULTI have different AIC result [closed]

I try to find the best model by using glmulti. Code and result below. So, I choose the simplest model, model 2. When I use glm ...
0
votes
0answers
16 views

Normal vs. non normal distributed data. Choice of model

I have a question about normally distributed and non normally distributed data. I am going to do a genotype association analysis but I want to build a model for each trait that I have to know which ...
0
votes
1answer
21 views

K-fold CV based model selection with a constraint on the number of features?

I am currently working on project where I need to train a logistic regression classifier with a combined $l_1$/$l_2$-penalty that satisfies a hard on the number of features. Specifically, my dataset ...
1
vote
1answer
20 views

Model selection in PGLS?

I am using Phylogenetic General Least Squares in the R package 'caper'. I have 4 predictor variables and I would like to know which are correlated with my response variable, while taking the ...
0
votes
1answer
29 views

ARMA model selection: in-sample vs. out-sample accuracy measures

I have a time series for 1000 days for many firms. I am interested to know, in general, on what basis I should select an ARMA model (the nature of my problem restricts integration order to 0). Should ...
0
votes
1answer
17 views

ARIMA Time series forecasting in R, help on choosing adequate model

I am trying to create adequate time series model in R. I have doubt about adequacy. My data is year and total number of events: ...
0
votes
0answers
15 views

Confusion over the test error and the expected error and actual implementation

Given a training set, $\tau = \{(x_1,y_1,\dots, x_N,y_N \}$ and a model $\hat{f}(x)$ has been fit. We have the following two definitions: The Generalisation (Test) Error $$ Err(\tau) = ...
0
votes
1answer
40 views

AIC, model selection and overfitting

I am looking for references that specifically show that Akaike's Information Criterion (AIC), or its corrected form (AICc), can in some practical applications -- that is, not in the asymptotic regime ...
3
votes
1answer
28 views

Is AIC a valid criterion for the selection of variance structures in GLS?

In generalized least squares, I’ve specified a weights function that accounts for heterogeneity in residual variance that exist along the range of a covariate. The validation graphs (residuals ...
1
vote
1answer
44 views

What comes first: outlier detection or model selection?

I'm fitting a GLMM (mixed logistic regression) in R. I have five covariates. For model selection, I'm using glmmLasso() (in R) to determine which of the five covariates and their interactions should ...
1
vote
0answers
9 views

Should I run variable selection within MICE for Multiple Imputation?

I have a dataset with around 100 variables, and I plan on getting multiply imputed datasets using the mice package in R. The literature I have read seems to advocate regressing each variable against ...
3
votes
0answers
12 views

PDF/CDF ratio interpretation

What would be the intuitive interpretation of the ratio of probability and cummulative density function during the first stage of Heckman selection model? Is it the likelihood of selection for the ...
0
votes
0answers
2 views

What component of the result should I look at when doing a LLM model fit?

I am running mixed effects models with poisson and negative binomial fits. To asses which of the models are better, what components of the models should I look at? Some popular methods I follow: a) ...
0
votes
0answers
13 views

range of possible values and model complexity

I have a prediction model where the likelihood is a function of two components $(A,B)$ $L \sim \bf{A}^\alpha + \bf{B}^\beta$ , where both $\bf{A}$ and $\bf{B}$ are $n\times n$ matrices ($n$ is the ...
0
votes
0answers
14 views

Cancelling roots in ARMA(1,1) with external regressors

I am trying to find out what cancelling roots would imply for the estimators of my external regressors in my ARMA(1,1) model. Unfortunately however I'm stuck in my final step since I'm insecure about ...
0
votes
0answers
32 views

Help discussing stationarity using correlograms? ARMA/ARIMA modelling

I am currently trying to understand how to use correlograms to examine stationarity and analysis the appropriate models. Please can you advice, below I have included my ACFs and PACFs, and I am trying ...
1
vote
0answers
16 views

Likelihood ratio test seems to show little difference between models with AICc difference of 3

I'm running a multinomial logistic regression analysis of the behavioural responses of deer to camera traps using no reaction, reaction and strong reaction as dependent variables and season, camera ...
0
votes
1answer
42 views

Using Information theory with all possible models to select the best Model

I have a simple data set to find out about the effect of cultivation period length on soil organisms. The main factor of interest is age_class, a categorical variable defining the age of a field under ...
3
votes
0answers
35 views

AIC equivalent to Mallows' Cp and Mallows' Cp unbiased for test MSE

Part 1: The goal is to show that with Gaussian errors and a linear model, Mallows' $C_p$ and $AIC$ are equivalent. Using our definition of Mallows' $C_p$: $$C_p=1/n(RSS+2d\hat\sigma^2)$$ and $AIC$: ...
1
vote
1answer
34 views

Comparing the distribution fits of a bivariate and a univariate model

Suppose I've done an experiment and I have a distribution of observations $x$ that vary between $-\pi$ and $\pi$. Now suppose each $x$ is associated with a second observation $y$ that may or may not ...
0
votes
1answer
34 views

Model-based clustering evaluation with BIC

Let's say I have fitted two models using EM-clustering and they differ in both the number of clusters and are fitted on different subset of features (chosen from the same set of features). Could I ...
0
votes
1answer
15 views

Can one use k-fold cv and holdout analysis together?

I would like to start by saying i have just started using cross-validation, so please bear with me if the questions seems very trivial. I am reviewing someones work where the person has used k-fold ...
4
votes
0answers
37 views

Comparing the performance of two classifiers using cross-validation

Consider the following excerpt (paraphrased, see sec. 4.6.3 for original wording) from Introduction to Data Mining (free chapter) by Pang-Ning Tan, Michael Steinbach, and Vipin Kumar. Suppose we ...
3
votes
2answers
38 views

How to make sure kernel density estimation has a proper data size?

I'm using cross validation with kernel density estimation. In cross validation, the dataset is divided into several fold, which would make the test dataset has smaller size. And I'm wondersing if ...
0
votes
0answers
70 views

Log-likelihood ratio test vs. information criteria for model selection

I am trying to select the model with the best fit among GARCH(1,1), ARMA(2,2) and GJR-GARCH(1,1) models for a time series of log returns. The results from IC (Akaike, Bayesian) and likelihood-ratio ...
0
votes
0answers
41 views

R^2 for mixed effect models (both generalized linear and additive)

I have seen from several discussion threads that there are a few ways of calculating R^2 for LMMs and GLMMs - albeit with a caveat for GLMM being that the existing methods work for only gaussian ...
0
votes
1answer
28 views

Using AICc distributions to assess goodness-of-fit and model selection

I have a couple of ordinary differential equation models that I'm trying to fit to time-dependent biological data ($y_n$). One model is more complex than then other as it has more free parameters. I ...
2
votes
0answers
18 views

GLM with exponential and linear terms (variable exposure)

Suppose I'm counting the number of cars of a particular type that pass by a spot. I drive out to the spot at some time (that may differ depending on the day), sit there counting the cars, and note the ...
1
vote
0answers
20 views

Does a check failing to compare observed and predicted data qualify as a posterior predictive check?

I consider a Gaussian mixture distribution and I want to implement posterior predictive checks for choosing the model with the correct number of mixture components. I know the true number of ...
1
vote
0answers
29 views

Should difference between accuracy of model on training data and testing data be considered for model selection?

Suppose I have two models (Model 1 and Model 2), Where Accuracy of Model 1 on test data is higher than that in Model 2 Difference between accuracy of model on test data and training data is higher ...
0
votes
1answer
41 views

What to check in cross-validation - MAE or MSE?

When using cross-validation for model selection, should one look at MSE or MAE. I know that MSE and MAE are related but which is the more appropriate measure?
0
votes
0answers
12 views

Cross-validation for model selection

When using cross-validation for model selection should does one have to use the same k-folds to train all the models. What is mean is if i have say an OLS model and i want to find out which ...
72
votes
2answers
4k views

How much do we know about p-hacking “in the wild”?

The phrase p-hacking (also: "data dredging", "snooping" or "fishing") refers to various kinds of statistical malpractice in which results become artificially statistically significant. There are many ...
0
votes
1answer
77 views

Time series model selection: AIC vs. out-of-sample SSE and their equivalence

AIC is frequently recommended as criterion to compare models for time series forecasting. See for example this in the context of dynamic regression models: The AIC can be calculated for the final ...
1
vote
0answers
14 views

Statistical models for explanation purposes?

I am familiar with many "black-box" models used for predictive purposes (e.g. random forests, xgboost, neural networks), however these do not explain which variables are most significant in explaining ...
1
vote
0answers
25 views

Model selection in ensembles

I'm trying to build an ensemble for a ML problem where fast prediction time is critical. So I'm interested in keeping my set of level-0 models for the ensemble pruned. Which measures can I use to ...
1
vote
0answers
15 views

Regressor selection using full subset selection and simulation error minimization

I hope my question is in line with the topic of this site. I would like to identify the set of correct regressors form the following equation, which is a kind of Henon map: $ y(k) = 1 - a \big( ...
3
votes
1answer
89 views

Model construction: How to build a meaningful gam model? (generalized additive model)

I have seen there are various questions concerning the interpretation and construction of gams, which seems to illustrate the difficulty for non-statisticians to deal with those. Unfortunately, from ...
1
vote
0answers
54 views

Is it possible that, for the smaller value of the cost parameter, the margin become small, while for the larger cost, the margin become larger?

I have written an R code for soft margin svm using the ipop function of kernlab package. Despite it's working fine, but I still have some doubt whether this code working properly or not. According to ...
2
votes
2answers
68 views

How to reduce the final set of significant variables from logistic model?

I have built a logistic model, which has 40 significant predictors, p value<0.0001. I want to reduce them to say about 10 variables, so that it can be presented to business. How do i go about doing ...
2
votes
0answers
10 views

Higher SIC and lower S.E. of residuals

I used the Schwarz Information Criterion (a.k.a., BIC) and the Akaike Information Criterion (AIC) to select the models for a time series Analysis. AIC got me an ARMA (5,4) and SIC got me ARMA (2,1). ...
3
votes
3answers
108 views

A question about the choice and interpretation of the jumping distribution in Metropolis-Hastings algorithm

In order to implement the MH algorithm you need a proposal density or jumping distribution $q(⋅|⋅)$, from which it is easy to sample. If you want to sample from a distribution $f(⋅)$, the MH algorithm ...