# Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

1,854 questions
Filter by
Sorted by
Tagged with
209 views

### Selecting ARIMA model based on PACF and ACF graphs

I am facing two pairs of ACF and PACF graphs. Unfortunately, I cannot use auto.arima, but I need to make sure my intuition is correct. 1. 2. 1. I believe that an ARIMA(1,1,0) would produce similar ...
105 views

### cross-validation and test set

Apologies if this has been answered before elsewhere. Answers I have read so far have only confused me further. Essentially, I want to check whether I can use the test set to choose betweeen two ...
1 vote
54 views

### How to determine the seasonal component order of an ARIMA model?

I am implementing two ARIMA models in R, I used a first order differencing and a seasonal differencing to get the stationary data, like this: For the first one I used auto.arima() and got an ARIMA(...
41 views

### Single or multiple model. How to know beforehand?

I am building a probability of default model based on behavioral information. The dataset is a loan portfolio, which contains 4 types of loans: mortgage, unsecured loans, car loans and credit cards. ...
25 views

### Dimension selection based on test accuracy

consider that I have a dataset with train, validation and test set and I want to train the pipeline PCA+logistic regression classifier. So far, for a specific k (that is the reduced data dimension ...
1 vote
32 views

### Why does the best fit model (lower AIC) yield higher p values than models with higher AIC? [closed]

Background: I am running a model selection in R that includes 1, 2, and 3-covariate models. Each model aims to determine the effect of environmental covariates in the occupancy of different species in ...
37 views

### Model/variable selection in binomial models for marble counts?

Thank you in advance for your help with this interesting question that has come up in my research. I changed the problem to bags/marbles to simplify. Problem description There are N bags, each ...
105 views

### Time Series Forecasting Process With Regard to Training and Test Sets

I'm a bit confused about the process order in doing proper time series analysis/forecasting. Is it: Stationary/seasonality checks, do any transformations required Candidate model selection using ACF, ...
1k views

### What if there is no true data-generating process?

I've been engaging in a number of forecasting efforts recently, and have rediscovered a well-known truth: That combinations of different forecasts are generally better than the forecasts themselves. ...
• 2,617
1 vote
49 views

### Should one use the usual splitting (Learning/Validation/Test) when using cross-validation?

Say you want to tune several parameters of your model using $N$ data. What you usually do is splitting your $N$ data into 3 sets: learning set: used to build your model; validation set: used to ...
• 161
72 views

### Why do we choose the hyperparameters that gives the lowest validation error? Do we assume that it also gives the lowest generalization error?

The usual way of selecting hyperparameters is to tune it on the validation set and select the hyperparameters that gives the lowest validation error (Lets assume the validation sample is large so we ...
• 179
34 views

### Determining Order of ARMA model

I'm working with time series data in R and I'm trying to figure out the order of an ARMA model. I'm not quite sure how to interpret the PACF and ACF plots. The series was proven stationary through an ...
1 vote
208 views

### VAR model selection: is AIC an appropriate measure given that the sample size changes depending on the number of lags?

Given that the sample size of a VAR (or a similar model: VARX, SVAR etc.) reduces by $1$ for each extra dependent-variable lag that I introduce (since we need to drop the empty rows, or ...
• 252
53 views

### Right kind of statistical analysis?

I'm performing a user study where we are attempting to compare two forms of measurement of participant state-of-mind. Both measurements are numerical scores - one is self-reported, and the other is ...
45 views

### Model complexity vs regularization

How do the complexity of a model and regularization behave with each other? Like we could decrease the degree of a polynomial or add a regularization term. Or both? In other words: Why is there ...
• 2,309
14 views

### Are tree based models for regression/classification 'endlessly' trainable?

I know this is a relatively simple question that I could answer if I understood trees more. An ANN can be trained indefinitely, especially if it is deep. What other models besides networks have this ...
1 vote
72 views

### VAR - VARX model selection

Suppose I have three stationary economic time series $y_i$ that are not cointegrated and I want to investigate the relationship between them. I happen to be unsure about the "endogenousness" ...
• 252
812 views

### Is it wrong to compare multiple models on the same test set and choose the best model?

Suppose we split a dataset into 3 parts (train, validation, and test). I know that it's important to make sure the test set doesn't influence our decisions during model selection or hyperparameter ...
34 views

### Lag choice for a VAR model

There is something confusing (for me) about this question. I'm working on a VAR model (7 time series) where i've checked for Granger causality (yes) and stationarity (yes). Now, according to the AIC ...
1 vote
88 views

### ARMA zero coefficients (not statistically significant)

I seem to have forgotten what to do when the coefficients of the time series model are not statistically significant like $\phi_2$, $\phi_3$, $\phi_4$ below. A university course taught us to set each ...
• 441
1 vote
54 views

### automatic model selection of linear models

https://i.stack.imgur.com/uYayd.gif https://i.stack.imgur.com/bAEMc.gif tail -c +43 uYayd.gif > TROW.tsv tail -c +43 bAEMc.gif > AABB.tsv Using the two ...
• 532
1 vote
92 views

### How can we assume the models are exhaustive in Bayesian Model Averaging?

Bayesian model averaging is justified using the law of total probability which requires the the set of models that we average over to be exhaustive. Shouldn’t we prove that the set of models are ...
132 views

### Interpretation of Elastic net having too low or high value of alpha

Often I found the situation that the elastic model what I fitted has optimal alpha value at 0 or 1. Or not only that situation, but also there some alphas go near to 0 or 1.(ex. 0.1 or 0.9) My ...
1 vote
604 views

### Thousands of features and only 70 samples

I am working on a regression problem where I have around 5-10 thousand features and have only 65 samples. I am training my algorithm on 55 samples and testing on 10 samples. I am using both Pearson ...
• 11
1 vote
375 views

• 133
283 views

### Neural network using Orange

I'm using Orange Data Mining in a regression analysis applying Artificial Neural Network (ANN). Some works suggest defining the number of neurons in the input layer as the number of variables. The ...
44 views

### Model fitting VS model selection: what works best?

Suppose we have two candidate models to predict a variable $y$ given a variable $x$, where $\alpha$ is a model parameter. $$\hat y=M_1(x,\alpha)$$ $$\hat y=M_2(x,\alpha)$$ Conceptually, we could ...
• 1,046
236 views

### R Squared (OOB) and R Square from correlation of prediction of test set is different?

I'm using simulated data and fitting Random Forest model for regression on a training dataset. What is confusing me is that after running Random Forest, I got R Squared (OOB) equal to 0.14. But when I ...
188 views

### Linear model selection (AIC/dredge function) with non parametric data (residuals are autocorrelated)

I used the dredge function with MuMIn & lme4 package to do a linear model selection with AIC principle. I have about 10 ...
117 views

### Is BIC asymptotically efficient for minimizing prediction error if the true model is being considered?

If a set of models is being compared using BIC and AIC, given the fact that the true model (the one which generated the data) is in this set (and given the other assumptions that guarantee BIC ...
112 views

### Fixed effects in DAGs

Let's imagine I'm interested in studying the causal effect of beliefs in some ideas and behavior related to these ideas (say, if I believe sunscreen is good for my health, I use more sunscreen etc.). ...
• 1,750
25 views

### Interprete AIC as significance test and determine significance level for nested models

Let's say we have two nested models. The smaller one (corresponds to $H_0$) has $p_0$ parameters, so its AIC is given by $$AIC_0=-2\log L_0+2p_0$$ The larger model has $p_1:=p_0+d$ parameters of which ...
• 101
20 views

### How can compare suggestion models with different performances?

I have 4 class binary classification models. That models identify which class a particular students is suitable for. For example, we have user 1 and 4 classes ...
• 101
1 vote
70 views

### Split one model into two models for a classification problem

In the classification problem, I am working on right now, I have to classify transactions mainly with text data. The classes of the training set can easily be divided into class sets with a negative ...
• 11
41 views

### Stabilizing nested cross-validation

Quite recently I stumbled upon several posts here on nested cross-validation, which showed me how wrong my understanding was of such procedure. Now, trying to put all the pieces together, I still have ...
• 1
48 views

### Can I use AIC for path models and non-path models?

I am testing competing hypotheses where one hypothesis contains a mediation effects that can be modeled using a path model. The other hypothesis does not include a mediation effect and therefore can ...
• 314
349 views

### MultiClass Classification - Training OvO and OvA

I like to know how OvO (One vs One) and OvA (One vs All) models are trained in multiclass classification problem. To keep it simple, we have 4 classes, each of which has 1000 datapoints. What are the ...
40 views

### ANOVA for comparing same linear models

I have trained two linear regressor models with the same response $Y$ and the same features $x_1$ and $x_2$, they are basically the same model, however only the training data differs: in model1 data ...
313 views

### How to determine the optimal lag order of asymmetric innovation for a GARCH?

In the python package arch_model, there is an option with which we can give the "lag order of the asymmetric innovation" when we estimate GARCH model. Are there any ways to find the optimal ...
• 539
2k views

### Why does the Akaike Information Criterion (AIC) sometimes favor an overfitted model?

As an exercise to develop practical experience working with model selection criteria, I computed fits of the highway mpg vs. engine displacement data from the tidyverse mpg example data set using ...
• 3,002