Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

Filter by
Sorted by
Tagged with
0 votes
1 answer
141 views

Parameter estimation of ARIMA model with exogenous variables (ARIMAX)

I am trying to compare an ARIMA model based on the price of a cryptocurrency without exogenous variables to one which adds in the number of tweets about the crypto in the same period as an exogenous ...
user avatar
0 votes
0 answers
149 views

Criteria for saving best model during training neural network?

I am doing 4-class semantic segmentation with U-net using generalised dice loss as loss function. General approach to save best model during training is to monitor validation loss at each epoch and ...
user avatar
  • 11
7 votes
3 answers
2k views

Why is AIC or BIC commonly used in model selections for time series forecasting?

On scikit-learn documentation, I found the following comments about AIC: Information-criterion based model selection is very fast, but it relies on a proper estimation of degrees of freedom, are ...
user avatar
  • 445
0 votes
0 answers
42 views

Orders of AR and MA models

I have a couple of ACFs and PACFs and cant seem to be sure about the order of AR(p) and MA(q).Can anybody kindly give me an insight into how to detect that?
user avatar
1 vote
1 answer
126 views

How to find the order for ARMA model?

I have a problem finding the order with the ACF and PACF plot, below is it. First I think they can be considered as tails off gradually because they are abnormal, then I set AR(1) from PACF and MA(1) ...
user avatar
  • 123
0 votes
1 answer
165 views

Does the column ordering matter in the stepwise algorithms used by R?

Suppose I have a large data set with variables $x_1, x_2, \ldots, x_p$ to predict response $y$ where $p$ is very large (however $n >> p$). I would like to perform forward stepwise regression on ...
user avatar
  • 175
3 votes
1 answer
1k views

Convergence issues and model selection in glmmTMB

Convergence problems in mixed effect models seem to be a common struggle. It is my understanding that they emerge when the likelihood surface is too flat for the optimisation algorithms to find a ...
user avatar
1 vote
2 answers
65 views

Is the linear mixed-effects model the right choice for analysing my data?

I'm in desperate need of advice in terms of the choice of statistical test for my analysis. Briefly to explain what I am analyzing in an animal model: I want to see the effect of 2 categorical ...
user avatar
1 vote
0 answers
112 views

Nested Cross validation with two settings on KNN

I am trying to perform model selection and evaluation using a 5-fold (internal) CV for the iris data. The things that I performed so far. Partitioned ...
user avatar
  • 259
1 vote
0 answers
16 views

Which model to select of two similarly performing, models with similar architecture and number of parameters, but different depths

I am training U-Net models for two-class semantic segmentation (foreground/background). I have tested different depths of the U-Net along with different number of filters in the first conv-layer (the ...
user avatar
  • 131
0 votes
0 answers
28 views

Selection of a “best” regression model using differet approaches

I need help with the following question. I am really lost, so any help/hint would be much appreciated! I am aware that for best fit model, we are looking for higher $R^2_{Adj}$, smaller $MS_{res}$, ...
user avatar
  • 133
0 votes
1 answer
18 views

Should we use the same fitting method in model selection and prediction?

I am curious whether we have to use the same fitting method in model selection and prediction. For example, suppose that we are going to use the logistic regression in prediction. Then, we may select ...
user avatar
  • 597
1 vote
0 answers
68 views

Can I always defend using ols, (for example when my dependent variable is ordinal), if I satisfy all CLM assumptions?

Previous reading Let me first say that I went through this post: (How to determine which distribution fits my data best?) and this post: Assumptions of linear models and what to do if the residuals ...
user avatar
  • 239
0 votes
1 answer
247 views

What are the preliminary analysis before running a logistic regression?

I have a dichotomous variable which represents if a student is accepted or not in a University. In order to do this I have about 60 variables (information of the students: gender, age, etc; their ...
user avatar
7 votes
1 answer
218 views

GAM: Do shrinkage smooth splines also address for concurvity?

I have a gam model with automatic predictor selection based on cubic splines (bs = cr) and SELECT == T or shrinkage cubic splines (bs = cs) and SELECT == F. Now I'm wondering if predictors affected ...
user avatar
1 vote
1 answer
66 views

ARIMA (p and q values)

I am trying to do Arima forecasting, i differenced once so d=1, Im not sure what my p and q values need to be, please check screenshots of acf and pacf below:
user avatar
  • 13
1 vote
1 answer
70 views

Are there any key reasons as to why people would choose between apARCH, gjrGARCH and E-GARCH?

I’ve been doing a lot of R coding with GARCH for my dissertation, I'm coming to the end of my writeup now but have hit a bit of a wall. Obviously, gjrGARCH, apARCH and E-GARCH all allow for asymmetric ...
user avatar
0 votes
0 answers
56 views

Identify ARIMA Model from ACF & PACF plots

I have plotted ACF & PACF plots for my data, but getting no significance on 1st lag and on further lags it is showing negative lag. please help to identify order of ARIMA model. Thank you
user avatar
0 votes
2 answers
82 views

How to select the best regression model from three models based on hypothesis testing

the following regression models were developed based on the same dataset: model 1: y=a1x1 + b1x2 + c1 model 2: y=a2x1 + b2x3 + c2 model 3: y=a3x1 + b3x4 + c3 where a, b, and c are the regression ...
user avatar
0 votes
0 answers
65 views

Picking a suitable performance metric when comparing the same model but using different sets of training data (Causal inference model)

I am comparing the same models prediction accuracy (Causal Impact) using different control variables as predictors and looking for a metric to decide which set of controls to use. Reading into AIC and ...
user avatar
1 vote
0 answers
269 views

A VECM in logs or a ARDL in the first difference of logs?

Suppose I have a number of time series that appear to have exponential growth at similar rates, with errors I believe to be generally proportional to the level of the variable. I believe that one of ...
user avatar
  • 2,617
2 votes
0 answers
1k views

Model smoothness selection for GAMs: GCV vs. REML vs. ML?

I am studying patterns of bird abundances with certain habitat variables and how they vary over time. I am interested in using GAMs with smooth terms for some of the variables. I am, however, confused ...
user avatar
0 votes
1 answer
31 views

Kfold model selection with high train test score deviance

For model hyper parameter tuning, standard k-fold cross validation is being performed. Scoring is done using coefficient of determination, with higher test scores preferred. Dataset size is (100K, 30)....
user avatar
  • 101
1 vote
0 answers
102 views

BIC and RMSE are Contradict each other for ARIMA Model Selection using R: Do I Err in Theory or in Practice?

I was surprised to see that RMSE and BIC have contradictory trends for the same time-series data. EDITED The procedures in my code are: simulate a 15 AR series of ...
user avatar
1 vote
1 answer
284 views

Singular fit for model and want to retain random intercept

I've read many posts about singular fit issues and this: https://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#singular-models-random-effect-variances-estimated-as-zero-or-correlations-estimated-as--...
user avatar
1 vote
1 answer
31 views

Parsimonious model - non convergent model decreases fit when removing random slope with near zero variance

I am trying to use Bates et al (2015) recommendation on reducing a maximal model based on variance explained by the random terms. Importantly, all examples in Bates are models with exclusively ...
user avatar
1 vote
1 answer
41 views

Comparison of conditional AIC (cAIC) from two or more LMMs

I have a crime data of 300 local divisions (level 1 units) unevenly distributed across 40 districts (level 2 units). I call it Data1. I decided to use cluster analysis to reclassify the level 1 units ...
user avatar
1 vote
0 answers
12 views

Impact of ACF and PACF plots on p,q,d in ARIMA and interpretation of these plots in Python?

I have these two plots: ACF, PACF and my question is: what is an interpretations of these plots and by looking on this plots what values of parameters p, q ,d should I choose for my ARIMA model ?
user avatar
  • 21
1 vote
0 answers
116 views

Model Selection for Multivariate Time Series Classification Problem

Problem Definition: You are given a dataset of $N$ different features. Most of the features are actually calculations of their own time series of set length (i.e A 5-time step weighted moving average)...
user avatar
1 vote
0 answers
84 views

What counts as a parameter for AIC?

I know this question has been asked before (e.g. here Meaning of 'number of parameters' in AIC), but I am still confused. What exactly makes something as a parameter for the AIC penalty, ...
user avatar
  • 121
1 vote
0 answers
35 views

Is the model with the lowest AIC value always the preferred one? [duplicate]

I have a question regarding model selection based on the AIC criterion. I have 5 predictor variables that I include in my models (no interactions are included) and create all possible model ...
user avatar
  • 11
4 votes
1 answer
85 views

Interview question: train/test error and "best" model

I recently had a puzzling interview question and I am wondering whether anybody can tell me the intended answer. The question shows train and test error for three models plotted against the number of ...
user avatar
  • 10.1k
1 vote
0 answers
45 views

ARIMA PACF plot interpretation

I have this dataset about voters in a country There is a clear trend therefore I differentiate the data once to achieve a stationary data The resulting acf and pacf plots are below Based on my ...
user avatar
1 vote
0 answers
72 views

Do conventional thresholds for global fit indices (e.g. AIC) hold for models based on very large data sets?

Problem/Question in short: I have estimated 5 generalized linear mixed models and subsequently compared their levels of relative fit according to AIC. These models are based on a very large dataset of ...
user avatar
  • 51
1 vote
1 answer
137 views

Multiple comparisons correction, for alpha-less criteria like AIC

When performing multiple hypothesis tests, for example in stepwise model selection, we need to apply something like the Bonferroni correction to the alpha/significance value in order to avoid too many ...
user avatar
  • 273
0 votes
0 answers
57 views

fit suitable AR-GARCH models to returns in MATLAB

I fit AR(1)-GARCH(1,1) models to 100 shares return, but at almost all of them the fitted AR(1) model is not stationary (AR{1}=0.99). How I can find a suitable same model? (for example by using the AIC ...
user avatar
  • 1
1 vote
1 answer
165 views

How can we select the best GARCH model by carrying out likelihood ratio test?

I have carried out the likelihood ratios of different GARCH models. GARCH(1,1) and GARCH(1,0)- Rejected null hypothesis so I chose GARCH(1,1) to do more sophistication. GARCH(3,1) and GARCH(1,1)- ...
user avatar
1 vote
1 answer
27 views

Which choice model to analyse my binary stated choice experiment data to estimate willingess-to-pay?

I followed the recipe of a stated-choice experiment in political science https://doi.org/10.1093/pan/mpt024 (they call it "conjoint" but I think this term is debated). In the end I made the ...
user avatar
1 vote
1 answer
100 views

Analyzing animal behavioural data in R Studio (GLMM) and struggling to understand how to set up the model

I'm trying to determine whether there was any differences in behaviour (count data, frequency per 30 minute observations, categorized into 5 categories [e.g., affiliative, agonistic, etc.]) of a ...
user avatar
1 vote
1 answer
161 views

Do you need a separate test set when performing nested cross-validation?

I understand that the inner cross-validation error can be biased and overly optimistic so that's why people use nested cross-validation to get unbiased generalization error. But we also have a hold ...
user avatar
0 votes
1 answer
45 views

cs231n: train/val accuracy and number of parameters

In this cs231 lection note there is a counterintuitive quote: The gap between the training and validation accuracy indicates the amount of overfitting. ... The other possible case is when the ...
user avatar
  • 126
0 votes
0 answers
90 views

Regression analysis with binary independent (categorical) variables and continous dependent variable

I am working on a dataset with 14 binary (0 or 1) independent variables (product features) and trying to measure a continous dependent variable (product price). I tried doing a regression analysis to ...
user avatar
4 votes
1 answer
81 views

Drawing the Line between a Model and a parameters in Bayesian Model selection

In Bayesian Model Selection, we first compute the evidence: $$ p(D|m) = \int_{\theta} p(D|\theta) p(\theta|m) d\theta $$ Then, we select the model that maximizes, which is a MAP estimator of the model....
user avatar
2 votes
0 answers
56 views

Comparing fits of linear models that use different data spans of a curve (and why does AIC seem to work)

Several related questions have been asked. This one is similar, but it does not match this question exactly. Also, i seem to have results that contradict the accepted answer there. Data An imperfect ...
user avatar
  • 41
0 votes
1 answer
168 views

ACF and PACF plots - MA, AR, ARIMA, or neither?

I differenced the variable (univariate model) in question because it was not stationary. I ran the dickey-fuller unit root test to check for stationarity, and it looks like the differenced version is ...
user avatar
0 votes
1 answer
112 views

How to find fitting ARMA-GARCH model? Financial data

I'm using financial data - logarithmic rates of return of WIG-Banks index, 2000 observations. I'm supposed to find ARMA-GARCH type of model, the most fitting one. Relying on ACF and PACF i estimated ...
user avatar
1 vote
0 answers
23 views

Linear model selection - Subset, Forward

We use 2**p model variations in order to define best model in subset selection. In contrary, we use 1 + p(p+1)/2 models in forward selection. But in the book it states, "Since we perform guided ...
user avatar
1 vote
1 answer
83 views

What is the difference between sieve estimation and structural risk minimization?

I was wondering if you could help me out. I am quite confused about the difference between sieve estimators (Ulf Grenander) and structural risk minimization (SRM) (Vladimir Vapnik). Could anyone give ...
user avatar
  • 111
2 votes
0 answers
28 views

Which model to select if paired-t-test do not show significance?

I have two classifiers: A and B. I have run 10-fold-cross-validation for both classifiers. A has better mean accuracy than B. Here are the scores: A: [0.82, 0.9, 0.88, 0.86, 0.88, 0.92, 0.84, 0.98, 0....
user avatar
  • 121
1 vote
0 answers
56 views

MA model or ARIMA model?

Anyone can give some suggestions to decide which model should i use for this time series forecast? It seems it is MA(1) by checking two plots? OR i can also try ARIMA model to compare each other? The ...
user avatar

1
3 4
5
6 7
38