# Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

1,854 questions
Filter by
Sorted by
Tagged with
141 views

### Parameter estimation of ARIMA model with exogenous variables (ARIMAX)

I am trying to compare an ARIMA model based on the price of a cryptocurrency without exogenous variables to one which adds in the number of tweets about the crypto in the same period as an exogenous ...
149 views

### Criteria for saving best model during training neural network?

I am doing 4-class semantic segmentation with U-net using generalised dice loss as loss function. General approach to save best model during training is to monitor validation loss at each epoch and ...
2k views

### Why is AIC or BIC commonly used in model selections for time series forecasting?

On scikit-learn documentation, I found the following comments about AIC: Information-criterion based model selection is very fast, but it relies on a proper estimation of degrees of freedom, are ...
42 views

### Orders of AR and MA models

I have a couple of ACFs and PACFs and cant seem to be sure about the order of AR(p) and MA(q).Can anybody kindly give me an insight into how to detect that?
1 vote
126 views

### How to find the order for ARMA model?

I have a problem finding the order with the ACF and PACF plot, below is it. First I think they can be considered as tails off gradually because they are abnormal, then I set AR(1) from PACF and MA(1) ...
165 views

### Does the column ordering matter in the stepwise algorithms used by R?

Suppose I have a large data set with variables $x_1, x_2, \ldots, x_p$ to predict response $y$ where $p$ is very large (however $n >> p$). I would like to perform forward stepwise regression on ...
1k views

### Convergence issues and model selection in glmmTMB

Convergence problems in mixed effect models seem to be a common struggle. It is my understanding that they emerge when the likelihood surface is too flat for the optimisation algorithms to find a ...
1 vote
65 views

### Is the linear mixed-effects model the right choice for analysing my data?

I'm in desperate need of advice in terms of the choice of statistical test for my analysis. Briefly to explain what I am analyzing in an animal model: I want to see the effect of 2 categorical ...
1 vote
112 views

### Nested Cross validation with two settings on KNN

I am trying to perform model selection and evaluation using a 5-fold (internal) CV for the iris data. The things that I performed so far. Partitioned ...
1 vote
16 views

### Which model to select of two similarly performing, models with similar architecture and number of parameters, but different depths

I am training U-Net models for two-class semantic segmentation (foreground/background). I have tested different depths of the U-Net along with different number of filters in the first conv-layer (the ...
28 views

### Selection of a “best” regression model using differet approaches

I need help with the following question. I am really lost, so any help/hint would be much appreciated! I am aware that for best fit model, we are looking for higher $R^2_{Adj}$, smaller $MS_{res}$, ...
18 views

### Should we use the same fitting method in model selection and prediction?

I am curious whether we have to use the same fitting method in model selection and prediction. For example, suppose that we are going to use the logistic regression in prediction. Then, we may select ...
1 vote
68 views

### Can I always defend using ols, (for example when my dependent variable is ordinal), if I satisfy all CLM assumptions?

Previous reading Let me first say that I went through this post: (How to determine which distribution fits my data best?) and this post: Assumptions of linear models and what to do if the residuals ...
247 views

### What are the preliminary analysis before running a logistic regression?

I have a dichotomous variable which represents if a student is accepted or not in a University. In order to do this I have about 60 variables (information of the students: gender, age, etc; their ...
218 views

### GAM: Do shrinkage smooth splines also address for concurvity?

I have a gam model with automatic predictor selection based on cubic splines (bs = cr) and SELECT == T or shrinkage cubic splines (bs = cs) and SELECT == F. Now I'm wondering if predictors affected ...
1 vote
66 views

### ARIMA (p and q values)

I am trying to do Arima forecasting, i differenced once so d=1, Im not sure what my p and q values need to be, please check screenshots of acf and pacf below:
1 vote
70 views

### Are there any key reasons as to why people would choose between apARCH, gjrGARCH and E-GARCH?

I’ve been doing a lot of R coding with GARCH for my dissertation, I'm coming to the end of my writeup now but have hit a bit of a wall. Obviously, gjrGARCH, apARCH and E-GARCH all allow for asymmetric ...
56 views

### Identify ARIMA Model from ACF & PACF plots

I have plotted ACF & PACF plots for my data, but getting no significance on 1st lag and on further lags it is showing negative lag. please help to identify order of ARIMA model. Thank you
82 views

### How to select the best regression model from three models based on hypothesis testing

the following regression models were developed based on the same dataset: model 1: y=a1x1 + b1x2 + c1 model 2: y=a2x1 + b2x3 + c2 model 3: y=a3x1 + b3x4 + c3 where a, b, and c are the regression ...
65 views

### Picking a suitable performance metric when comparing the same model but using different sets of training data (Causal inference model)

I am comparing the same models prediction accuracy (Causal Impact) using different control variables as predictors and looking for a metric to decide which set of controls to use. Reading into AIC and ...
1 vote
269 views

### A VECM in logs or a ARDL in the first difference of logs?

Suppose I have a number of time series that appear to have exponential growth at similar rates, with errors I believe to be generally proportional to the level of the variable. I believe that one of ...
1k views

### Model smoothness selection for GAMs: GCV vs. REML vs. ML?

I am studying patterns of bird abundances with certain habitat variables and how they vary over time. I am interested in using GAMs with smooth terms for some of the variables. I am, however, confused ...
31 views

### Kfold model selection with high train test score deviance

For model hyper parameter tuning, standard k-fold cross validation is being performed. Scoring is done using coefficient of determination, with higher test scores preferred. Dataset size is (100K, 30)....
1 vote
102 views

### BIC and RMSE are Contradict each other for ARIMA Model Selection using R: Do I Err in Theory or in Practice?

I was surprised to see that RMSE and BIC have contradictory trends for the same time-series data. EDITED The procedures in my code are: simulate a 15 AR series of ...
1 vote
284 views

1 vote
31 views

### Parsimonious model - non convergent model decreases fit when removing random slope with near zero variance

I am trying to use Bates et al (2015) recommendation on reducing a maximal model based on variance explained by the random terms. Importantly, all examples in Bates are models with exclusively ...
1 vote
41 views

### Comparison of conditional AIC (cAIC) from two or more LMMs

I have a crime data of 300 local divisions (level 1 units) unevenly distributed across 40 districts (level 2 units). I call it Data1. I decided to use cluster analysis to reclassify the level 1 units ...
1 vote
12 views

### Impact of ACF and PACF plots on p,q,d in ARIMA and interpretation of these plots in Python?

I have these two plots: ACF, PACF and my question is: what is an interpretations of these plots and by looking on this plots what values of parameters p, q ,d should I choose for my ARIMA model ?
1 vote
116 views

### Model Selection for Multivariate Time Series Classification Problem

Problem Definition: You are given a dataset of $N$ different features. Most of the features are actually calculations of their own time series of set length (i.e A 5-time step weighted moving average)...
1 vote
84 views

### What counts as a parameter for AIC?

I know this question has been asked before (e.g. here Meaning of 'number of parameters' in AIC), but I am still confused. What exactly makes something as a parameter for the AIC penalty, ...
1 vote
35 views

### Is the model with the lowest AIC value always the preferred one? [duplicate]

I have a question regarding model selection based on the AIC criterion. I have 5 predictor variables that I include in my models (no interactions are included) and create all possible model ...
85 views

### Interview question: train/test error and "best" model

I recently had a puzzling interview question and I am wondering whether anybody can tell me the intended answer. The question shows train and test error for three models plotted against the number of ...
1 vote
45 views

### ARIMA PACF plot interpretation

I have this dataset about voters in a country There is a clear trend therefore I differentiate the data once to achieve a stationary data The resulting acf and pacf plots are below Based on my ...
1 vote
72 views

### Do conventional thresholds for global fit indices (e.g. AIC) hold for models based on very large data sets?

Problem/Question in short: I have estimated 5 generalized linear mixed models and subsequently compared their levels of relative fit according to AIC. These models are based on a very large dataset of ...
1 vote
137 views

### Multiple comparisons correction, for alpha-less criteria like AIC

When performing multiple hypothesis tests, for example in stepwise model selection, we need to apply something like the Bonferroni correction to the alpha/significance value in order to avoid too many ...
57 views

### fit suitable AR-GARCH models to returns in MATLAB

I fit AR(1)-GARCH(1,1) models to 100 shares return, but at almost all of them the fitted AR(1) model is not stationary (AR{1}=0.99). How I can find a suitable same model? (for example by using the AIC ...
1 vote
165 views

### How can we select the best GARCH model by carrying out likelihood ratio test?

I have carried out the likelihood ratios of different GARCH models. GARCH(1,1) and GARCH(1,0)- Rejected null hypothesis so I chose GARCH(1,1) to do more sophistication. GARCH(3,1) and GARCH(1,1)- ...
1 vote
27 views

### Which choice model to analyse my binary stated choice experiment data to estimate willingess-to-pay?

I followed the recipe of a stated-choice experiment in political science https://doi.org/10.1093/pan/mpt024 (they call it "conjoint" but I think this term is debated). In the end I made the ...
1 vote
100 views

### Analyzing animal behavioural data in R Studio (GLMM) and struggling to understand how to set up the model

I'm trying to determine whether there was any differences in behaviour (count data, frequency per 30 minute observations, categorized into 5 categories [e.g., affiliative, agonistic, etc.]) of a ...
1 vote
161 views

### Do you need a separate test set when performing nested cross-validation?

I understand that the inner cross-validation error can be biased and overly optimistic so that's why people use nested cross-validation to get unbiased generalization error. But we also have a hold ...
45 views

### cs231n: train/val accuracy and number of parameters

In this cs231 lection note there is a counterintuitive quote: The gap between the training and validation accuracy indicates the amount of overfitting. ... The other possible case is when the ...
90 views

### Regression analysis with binary independent (categorical) variables and continous dependent variable

I am working on a dataset with 14 binary (0 or 1) independent variables (product features) and trying to measure a continous dependent variable (product price). I tried doing a regression analysis to ...
81 views

### Drawing the Line between a Model and a parameters in Bayesian Model selection

In Bayesian Model Selection, we first compute the evidence: $$p(D|m) = \int_{\theta} p(D|\theta) p(\theta|m) d\theta$$ Then, we select the model that maximizes, which is a MAP estimator of the model....
56 views

### Comparing fits of linear models that use different data spans of a curve (and why does AIC seem to work)

Several related questions have been asked. This one is similar, but it does not match this question exactly. Also, i seem to have results that contradict the accepted answer there. Data An imperfect ...
168 views

### ACF and PACF plots - MA, AR, ARIMA, or neither?

I differenced the variable (univariate model) in question because it was not stationary. I ran the dickey-fuller unit root test to check for stationarity, and it looks like the differenced version is ...
112 views

### How to find fitting ARMA-GARCH model? Financial data

I'm using financial data - logarithmic rates of return of WIG-Banks index, 2000 observations. I'm supposed to find ARMA-GARCH type of model, the most fitting one. Relying on ACF and PACF i estimated ...
1 vote
23 views

### Linear model selection - Subset, Forward

We use 2**p model variations in order to define best model in subset selection. In contrary, we use 1 + p(p+1)/2 models in forward selection. But in the book it states, "Since we perform guided ...
1 vote
83 views

### What is the difference between sieve estimation and structural risk minimization?

I was wondering if you could help me out. I am quite confused about the difference between sieve estimators (Ulf Grenander) and structural risk minimization (SRM) (Vladimir Vapnik). Could anyone give ...