# Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

1,463 questions
Filter by
Sorted by
Tagged with
8 views

### How to estimate the order of the ARDL model in R?

I have to build the best fitting ARDL model with d(log(GDP)) as the dependent variable and d(int. rate) as a regressor and use AIC for the lag selection with maximum 12 lags for the regressor and 12 ...
22 views

### Proving an alternative form for Mallow's $C_p$ statistic

I want to show that Mallow's $C_p$ statistic, $$C{_p}{_j} = \frac{RSS_j}{S_E^2} + 2s_j-n$$ can be written as $$C{_p}{_j} = (k+1-s_j)(F_j-1)+s_j$$ where $RSS_j$ is the residual sum of squares for model ...
25 views

### Can R^2 be used as the metric for model/parameter selection (grid search)?

I have a simple model with 3 free parameters to fit, and wanted to try estimating those parameters via grid search. I was wondering -- what is the convention for selecting the best model in this case? ...
29 views

### Guessing ARMA order just from the plot

I have this two plots, each one contains two realisations (orange/blue) of the same $ARMA(pi, qi)$ model. All orange instances share the same noise sequence $e_i$, and so do all the blue ones. I don'...
27 views

### Model selection based on the closest fit to another model

For each subject I have two measures, let's say $X$ and $Y$, that correspond to the same measurement but each from a different measurement device. In theory we assume that ideally these two ...
23 views

### Lag selection for Newey-West estimation in a panel data framework with Stata

I have a panel dataset with N=21 countries and T=8 two-year periods which are mechanically correlated (2010-2011, 2011-2012, 2012-2013 and so on...). Given this structure, I thought I should have ...
46 views

### How to compare nonlinear regression vs piecewise linear regression?

We have time series of plant growth (length vs time) and we can reasonably model these with two approaches: With a nonlinear model length(time)=length0*exp(r*time) where length0 is lenght at initial ...
20 views

### What are the steps and correct order of the operations in Machine Learning? [from Getting data to optimising models]

I've followed lots of tutorials on Machine Learning but in each of these, they go for a different strategy so it's quite confusing for me. I want to Know that what are the operations involved and what ...
8 views

### Model Selection for 'Groups' of Predictors

I have a dataset with one response variable and many predictor variables. I want to fit a "best" model to this dataset with one set of constraints; namely, the predictors all belong to one (and only ...
11 views

### Elastic Net: diverging number of parameters

I am reading the paper On The Adaptive Elastic-Net With a Diverging Number of Parameters by Zou and Zhang (2009). I found it while I was researching the lasso and elastic net in general and I am ...
28 views

### Compare different models via p-value, AIC and BIC

A sensor with the response Sw shall be investigated if it is affected by external influences like Temperature Tu and relative ...
28 views

### How to select variables when using shrinkage priors?

I am fitting a linear regression model using shrinkage priors (Horseshoe and Laplace/LASSO). This shrinks many of the variables close to zero, but I would like to select the variables. Can I use the ...
6 views

### Sample size calculation for variable selection

I know how to sample size calculation, for t-test, F-test or other tests which I know the distribution of the variable beforehand. But, I want to know how to calculate sample size for variable ...
49 views

### How to choose an appropriate method to fit this data?

I want to fit a model curve to the plot below. Each fine curve corresponds to a vehicle driving a certain distance and measureing some quantities of interest in certain timesteps (discrete ...
9 views

### A dramatic drop in AIC/BIC values (positive to negative) for a certain solution [duplicate]

I was using Stata running latent profile analysis and got fit indices as follows: AIC 221.25 / -643.82 / 237.25 and BIC 300.94 / -541.36 / 362.47 respectively for 3/4/5-profile solutions. Is there ...
14 views

### Minimizing AIC for VAR Model Results in Many Insignificant Lags

I have a pair of cointegrated ETF's. Both ETF's track the exact same underlying. SPY and IVV. I am using the vars package in R to build a VAR model and eventually a VECM. When I select the lags ...
23 views

### ABC SMC: How do weights scale proportionally with number of parameters

Having some problems with the ABC SMC algorithm. I'm trying to implement the methods taken from here: Simulation-based model selection for dynamical systems in systems and population biology How do ...
8 views

### Quasi-generalized least squares and model selection with SAS or R

I want to conduct a model selection (stepwise) on a linear model, in which the parameters are estimated using the quasi-generalized least squares (due to the presence of heteroskedasticity). Does a ...
15 views

### Decision tree with weights trained using RandomizedSearchCV - do I have to refit?

I trained a decision tree with weights using RandomizedSearchCV: ...
22 views

### Picking any model from RandomizedSearchCV [closed]

I trained set of models using RandomizedSearchCV and picked the best using .best_estimator_ and then tested on my test set. However, I would like to check how any other model from the grid performed ...
25 views

### State of the art (2019) of spike and slab priors?

Spike and slab priors for variable selection https://www.tandfonline.com/doi/abs/10.1080/01621459.1993.10476353 were initially proposed using a combination of normals prior and a mass at zero. After ...
11 views

### Machine learning model for classification of transactions

I'm very much a beginner in machine learning but I have a project that I need to complete. I have a large dataset of transactions (40 columns or more of attributes) and I need to classify them them ...
32 views

### What is the correct model (AR, MA, or ARMA) and order for the data?

I am new to time series and forecasting and I have been assigned to determine the model and order for a data object. The ACF, PACF, and EACF are below: I was thinking it was an AR(1), but I am not ...
6 views

### Model fit metrics for bayesian hierarchical models?

I have a hierarchical model that shrinks my individual now non independent models closer to the group level estimates. This is beneficial as it makes the inferences more robust. I have been using WAIC ...
31 views

### Dealing with a statistically insignificant variable while AIC decreases when introduced in the survival analysis model selection?

I am new to survival analysis and currently very confused regarding a model selection process (forward stepwise) in cox proportional hazard model. What happens is basically the following step by ...
21 views

### Clarifying some heuristics on choosing Neural Network hidden units

I'm looking to clarify a couple of rules of thumb for exploring neural network architectures - specifically, choosing the number of hidden units (in a bi-directional LSTM). Disclaimer: I know at the ...
8 views

### What causes a chol.default(tmpvc) error in the vuong test when comparing non nested models?

I have a series of ordinal logistic regressions. Each predicts the same outcome (Y), and each has a single predictor (X1, X2 or X3 (All are strongly correlated)). I want to determine whether any of ...
17 views

### best subset of moderators for meta-analysis in R

I have a large number of "categorical" moderators (35 moderators). I am planning to use the best subset of these moderators that can maximally explain the variation in my 257 correlated effect sizes ...
68 views

### Chosing the right time series model

Can someone help me to find the right time series model. I am not able to figure out the right hyper parameters for the model. Please see attached ACF and PACF graphs. I can provide more information ...
11 views

### About degree of freedom and model choice

I am re-examining an old problem of genetics. Between 1910 and 1925, the genetic model accepted for the AB0 blood groups (A, B, AB, and 0) was that there were two loci, L1 and L2, each with two ...
26 views

### when selecting an arima model, shouldn't we consider the characteristic roots as well as the AIC?

the Hyndman-Khandakar algorithm for automatic ARIMA modelling searches the model space and looks only on the AICc as its criteria for finding the best model. Shouldn't it also look at the ...
84 views

### Do I need a second validation set to select model class?

I want to choose a model class (e.g. logistic regression vs. random forests), but the validation set is used for selecting hyperparameters. Should I set aside a second validation set to select the ...
44 views

### How to compare/choose regression models based on predictions and prediction intervals

I have two regression models M1 and M2, each delivers a prediction (blue circle) together with a prediction interval as shown in blue in the picture below. The actual value is marked with a black ...
182 views

### When should I use Validation rather than Cross Validation

I am aware that CV was born as a way to validate models when there is a lack of training data, but my understanding is that it is generally better to cross validate rather than just use one validation ...
17 views

### Is there an AIC and BIC equivalent for MAP?

I would like to know if there is an AIC or BIC equivalent for maximum a posterior estimation. I'm trying to compare several different models that have been fit using MAP, but I am unsure of the best ...
24 views

### Nested Cross-Validation vs. Split-Sample Validation With a High n:p Ratio

With a high sample:predictor (n:p) ratio, as opposed to nested CV, why not just go with the split sample approach in which CV is done on training data (e.g., 80%) for model selection and estimation of ...
37 views

### Nested Cross Validation - Which Models Should We Evaluate in the Outer Loop?

Lets assume for example that I am attempting to predict a binary outcome using p predictors in which n>p with methods including a LASSO Regression, a Logistic Regression and SVM with an RBF kernel. ...
25 views

### Nested Cross Validation - How to Improve Models Without Bias

With nested CV, how can we get a sense of how our model may perform in the outer loop before actually bringing it to the outer loop? Without nested CV, if I did simple 10-Fold CV on training data (...
10 views

### What do model selection criteria actually mean? E.g. for generalization

I'm learning now about the various information criteria (AIC, BIC, TIC, WAIC,...) used in model selection. But I feel like I have a basic conceptual stumbling block. Suppose I have already collected ...
41 views

### Time Series Analysis - ARIMA

I am trying to make sales prediction from time series data. After performing a log transformation to the original data and differencing it by 1, I got a stationary dataset. So I plotted ACF and PACF ...
36 views

### Comparing AUC, logloss and accuracy scores between models

I have the following evaluation metrics on the test set, after running 6 models for a binary classification problem: ...
48 views

### Using AIC/BIC within cross-validation for likelihood based loss functions

For a course I am teaching, I am having my students fit a Gaussian mixture model using MLEs via the EM algorithm to a bivariate dataset. I have asked the students to use use cross-validation to choose ...
45 views

### Transformed data due to non-normal residuals - how to see if it actually improved the model?

I am trying to run a linear regression model (ideally) to see whether age (continuous variable) affects levels of stress hormone (also continuous, dependent variable), i.e. hypothesis testing. My ...
52 views

### understanding overfitting and underfitting and model selection

Is it true that Over-fitting yields a low error rate on the validation set? I'm a bit confused here. By definition, Overfitting means we have minimized the cost function with a given hyperparameter ...
10 views

### Which covariate is significant in a regularized GLM?

Let’s say we have 10 covariates ($X_1$, $X_2$, …, $X_{10}$) and we want to test how important $X_2$ is in predicting our neuron’s response ($Y$). What we do now is to compare two models (nested models)...
21 views

### Model selection based on confidence interval width

I'm looking for some papers which use model selection based on confidence interval width. ie best model is the one which has smallest sum of widths for given x. Are there any papers that use this or a ...
26 views

### How to do principal component regression for repeated measures?

I have between-subject repeated measures data. I want to select the parameters for my regression model. PCR /PCA is one option to reduce dimensionality. How can I do this in the case of a repeated ...
54 views

### When to use AR and when to use MA model?

When to use an AR model and when to use an MA model to model time-series data. What aspects of data are modelled by the AR process which can't be done by MA and vice-versa?