Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

Filter by
Sorted by
Tagged with
1
vote
0answers
8 views

How to estimate the order of the ARDL model in R?

I have to build the best fitting ARDL model with d(log(GDP)) as the dependent variable and d(int. rate) as a regressor and use AIC for the lag selection with maximum 12 lags for the regressor and 12 ...
0
votes
0answers
22 views

Proving an alternative form for Mallow's $C_p$ statistic

I want to show that Mallow's $C_p$ statistic, $$C{_p}{_j} = \frac{RSS_j}{S_E^2} + 2s_j-n$$ can be written as $$C{_p}{_j} = (k+1-s_j)(F_j-1)+s_j$$ where $RSS_j$ is the residual sum of squares for model ...
1
vote
0answers
25 views

Can R^2 be used as the metric for model/parameter selection (grid search)?

I have a simple model with 3 free parameters to fit, and wanted to try estimating those parameters via grid search. I was wondering -- what is the convention for selecting the best model in this case? ...
1
vote
0answers
29 views

Guessing ARMA order just from the plot

I have this two plots, each one contains two realisations (orange/blue) of the same $ARMA(pi, qi)$ model. All orange instances share the same noise sequence $e_i$, and so do all the blue ones. I don'...
0
votes
1answer
27 views

Model selection based on the closest fit to another model

For each subject I have two measures, let's say $X$ and $Y$, that correspond to the same measurement but each from a different measurement device. In theory we assume that ideally these two ...
0
votes
0answers
23 views

Lag selection for Newey-West estimation in a panel data framework with Stata

I have a panel dataset with N=21 countries and T=8 two-year periods which are mechanically correlated (2010-2011, 2011-2012, 2012-2013 and so on...). Given this structure, I thought I should have ...
2
votes
1answer
46 views

How to compare nonlinear regression vs piecewise linear regression?

We have time series of plant growth (length vs time) and we can reasonably model these with two approaches: With a nonlinear model length(time)=length0*exp(r*time) where length0 is lenght at initial ...
0
votes
1answer
20 views

What are the steps and correct order of the operations in Machine Learning? [from Getting data to optimising models]

I've followed lots of tutorials on Machine Learning but in each of these, they go for a different strategy so it's quite confusing for me. I want to Know that what are the operations involved and what ...
2
votes
0answers
8 views

Model Selection for 'Groups' of Predictors

I have a dataset with one response variable and many predictor variables. I want to fit a "best" model to this dataset with one set of constraints; namely, the predictors all belong to one (and only ...
1
vote
0answers
11 views

Elastic Net: diverging number of parameters

I am reading the paper On The Adaptive Elastic-Net With a Diverging Number of Parameters by Zou and Zhang (2009). I found it while I was researching the lasso and elastic net in general and I am ...
1
vote
1answer
28 views

Compare different models via p-value, AIC and BIC

A sensor with the response Sw shall be investigated if it is affected by external influences like Temperature Tu and relative ...
0
votes
1answer
28 views

How to select variables when using shrinkage priors?

I am fitting a linear regression model using shrinkage priors (Horseshoe and Laplace/LASSO). This shrinks many of the variables close to zero, but I would like to select the variables. Can I use the ...
0
votes
0answers
6 views

Sample size calculation for variable selection

I know how to sample size calculation, for t-test, F-test or other tests which I know the distribution of the variable beforehand. But, I want to know how to calculate sample size for variable ...
1
vote
1answer
49 views

How to choose an appropriate method to fit this data?

I want to fit a model curve to the plot below. Each fine curve corresponds to a vehicle driving a certain distance and measureing some quantities of interest in certain timesteps (discrete ...
0
votes
0answers
9 views

A dramatic drop in AIC/BIC values (positive to negative) for a certain solution [duplicate]

I was using Stata running latent profile analysis and got fit indices as follows: AIC 221.25 / -643.82 / 237.25 and BIC 300.94 / -541.36 / 362.47 respectively for 3/4/5-profile solutions. Is there ...
0
votes
0answers
14 views

Minimizing AIC for VAR Model Results in Many Insignificant Lags

I have a pair of cointegrated ETF's. Both ETF's track the exact same underlying. SPY and IVV. I am using the vars package in R to build a VAR model and eventually a VECM. When I select the lags ...
1
vote
0answers
23 views

ABC SMC: How do weights scale proportionally with number of parameters

Having some problems with the ABC SMC algorithm. I'm trying to implement the methods taken from here: Simulation-based model selection for dynamical systems in systems and population biology How do ...
0
votes
0answers
8 views

Quasi-generalized least squares and model selection with SAS or R

I want to conduct a model selection (stepwise) on a linear model, in which the parameters are estimated using the quasi-generalized least squares (due to the presence of heteroskedasticity). Does a ...
0
votes
0answers
15 views

Decision tree with weights trained using RandomizedSearchCV - do I have to refit?

I trained a decision tree with weights using RandomizedSearchCV: ...
0
votes
1answer
22 views

Picking any model from RandomizedSearchCV [closed]

I trained set of models using RandomizedSearchCV and picked the best using .best_estimator_ and then tested on my test set. However, I would like to check how any other model from the grid performed ...
0
votes
0answers
25 views

State of the art (2019) of spike and slab priors?

Spike and slab priors for variable selection https://www.tandfonline.com/doi/abs/10.1080/01621459.1993.10476353 were initially proposed using a combination of normals prior and a mass at zero. After ...
-1
votes
0answers
11 views

Machine learning model for classification of transactions

I'm very much a beginner in machine learning but I have a project that I need to complete. I have a large dataset of transactions (40 columns or more of attributes) and I need to classify them them ...
0
votes
1answer
32 views

What is the correct model (AR, MA, or ARMA) and order for the data?

I am new to time series and forecasting and I have been assigned to determine the model and order for a data object. The ACF, PACF, and EACF are below: I was thinking it was an AR(1), but I am not ...
0
votes
0answers
6 views

Model fit metrics for bayesian hierarchical models?

I have a hierarchical model that shrinks my individual now non independent models closer to the group level estimates. This is beneficial as it makes the inferences more robust. I have been using WAIC ...
1
vote
1answer
31 views

Dealing with a statistically insignificant variable while AIC decreases when introduced in the survival analysis model selection?

I am new to survival analysis and currently very confused regarding a model selection process (forward stepwise) in cox proportional hazard model. What happens is basically the following step by ...
1
vote
0answers
21 views

Clarifying some heuristics on choosing Neural Network hidden units

I'm looking to clarify a couple of rules of thumb for exploring neural network architectures - specifically, choosing the number of hidden units (in a bi-directional LSTM). Disclaimer: I know at the ...
0
votes
0answers
8 views

What causes a chol.default(tmpvc) error in the vuong test when comparing non nested models?

I have a series of ordinal logistic regressions. Each predicts the same outcome (Y), and each has a single predictor (X1, X2 or X3 (All are strongly correlated)). I want to determine whether any of ...
0
votes
0answers
17 views

best subset of moderators for meta-analysis in R

I have a large number of "categorical" moderators (35 moderators). I am planning to use the best subset of these moderators that can maximally explain the variation in my 257 correlated effect sizes ...
1
vote
1answer
68 views

Chosing the right time series model

Can someone help me to find the right time series model. I am not able to figure out the right hyper parameters for the model. Please see attached ACF and PACF graphs. I can provide more information ...
0
votes
0answers
11 views

About degree of freedom and model choice

I am re-examining an old problem of genetics. Between 1910 and 1925, the genetic model accepted for the AB0 blood groups (A, B, AB, and 0) was that there were two loci, L1 and L2, each with two ...
0
votes
1answer
26 views

when selecting an arima model, shouldn't we consider the characteristic roots as well as the AIC?

the Hyndman-Khandakar algorithm for automatic ARIMA modelling searches the model space and looks only on the AICc as its criteria for finding the best model. Shouldn't it also look at the ...
8
votes
2answers
84 views

Do I need a second validation set to select model class?

I want to choose a model class (e.g. logistic regression vs. random forests), but the validation set is used for selecting hyperparameters. Should I set aside a second validation set to select the ...
2
votes
1answer
44 views

How to compare/choose regression models based on predictions and prediction intervals

I have two regression models M1 and M2, each delivers a prediction (blue circle) together with a prediction interval as shown in blue in the picture below. The actual value is marked with a black ...
8
votes
2answers
182 views

When should I use Validation rather than Cross Validation

I am aware that CV was born as a way to validate models when there is a lack of training data, but my understanding is that it is generally better to cross validate rather than just use one validation ...
0
votes
0answers
17 views

Is there an AIC and BIC equivalent for MAP?

I would like to know if there is an AIC or BIC equivalent for maximum a posterior estimation. I'm trying to compare several different models that have been fit using MAP, but I am unsure of the best ...
0
votes
1answer
24 views

Nested Cross-Validation vs. Split-Sample Validation With a High n:p Ratio

With a high sample:predictor (n:p) ratio, as opposed to nested CV, why not just go with the split sample approach in which CV is done on training data (e.g., 80%) for model selection and estimation of ...
1
vote
1answer
37 views

Nested Cross Validation - Which Models Should We Evaluate in the Outer Loop?

Lets assume for example that I am attempting to predict a binary outcome using p predictors in which n>p with methods including a LASSO Regression, a Logistic Regression and SVM with an RBF kernel. ...
1
vote
1answer
25 views

Nested Cross Validation - How to Improve Models Without Bias

With nested CV, how can we get a sense of how our model may perform in the outer loop before actually bringing it to the outer loop? Without nested CV, if I did simple 10-Fold CV on training data (...
0
votes
0answers
10 views

What do model selection criteria actually mean? E.g. for generalization

I'm learning now about the various information criteria (AIC, BIC, TIC, WAIC,...) used in model selection. But I feel like I have a basic conceptual stumbling block. Suppose I have already collected ...
0
votes
1answer
41 views

Time Series Analysis - ARIMA

I am trying to make sales prediction from time series data. After performing a log transformation to the original data and differencing it by 1, I got a stationary dataset. So I plotted ACF and PACF ...
0
votes
0answers
36 views

Comparing AUC, logloss and accuracy scores between models

I have the following evaluation metrics on the test set, after running 6 models for a binary classification problem: ...
1
vote
1answer
48 views

Using AIC/BIC within cross-validation for likelihood based loss functions

For a course I am teaching, I am having my students fit a Gaussian mixture model using MLEs via the EM algorithm to a bivariate dataset. I have asked the students to use use cross-validation to choose ...
0
votes
2answers
45 views

Transformed data due to non-normal residuals - how to see if it actually improved the model?

I am trying to run a linear regression model (ideally) to see whether age (continuous variable) affects levels of stress hormone (also continuous, dependent variable), i.e. hypothesis testing. My ...
0
votes
1answer
52 views

understanding overfitting and underfitting and model selection

Is it true that Over-fitting yields a low error rate on the validation set? I'm a bit confused here. By definition, Overfitting means we have minimized the cost function with a given hyperparameter ...
0
votes
0answers
10 views

Which covariate is significant in a regularized GLM?

Let’s say we have 10 covariates ($X_1$, $X_2$, …, $X_{10}$) and we want to test how important $X_2$ is in predicting our neuron’s response ($Y$). What we do now is to compare two models (nested models)...
0
votes
0answers
21 views

Model selection based on confidence interval width

I'm looking for some papers which use model selection based on confidence interval width. ie best model is the one which has smallest sum of widths for given x. Are there any papers that use this or a ...
2
votes
0answers
26 views

How to do principal component regression for repeated measures?

I have between-subject repeated measures data. I want to select the parameters for my regression model. PCR /PCA is one option to reduce dimensionality. How can I do this in the case of a repeated ...
3
votes
1answer
54 views

When to use AR and when to use MA model?

When to use an AR model and when to use an MA model to model time-series data. What aspects of data are modelled by the AR process which can't be done by MA and vice-versa?
1
vote
0answers
22 views

What is the meaning of non-nested model in Vuong's test in R?

I am having hard time understanding the use of nested and non-nested models. According to Ben-Akiva's description the nested models are blue bus/red bus problems, where the choice set has multi ...
0
votes
1answer
60 views

Optimal lag-selection in VAR-model in R

Having troubles with the lag specification of a VAR-model. The purpose of the model is to measure orthogonal impulse/response function of oil price shocks on macroeconomic variables, such as GDP-...