Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

Filter by
Sorted by
Tagged with
0
votes
0answers
18 views

Selection of a “best” regression model using differet approaches

I need help with the following question. I am really lost, so any help/hint would be much appreciated! I am aware that for best fit model, we are looking for higher $R^2_{Adj}$, smaller $MS_{res}$, ...
0
votes
1answer
10 views

Should we use the same fitting method in model selection and prediction?

I am curious whether we have to use the same fitting method in model selection and prediction. For example, suppose that we are going to use the logistic regression in prediction. Then, we may select ...
0
votes
0answers
5 views

Subset Selection Methods for a Binary Response

The situation I face is as follows: 500 observations 100 predictors (features) A binary response What I want to do: Identifying a subset of the predictors that I believe to be related to the ...
1
vote
0answers
30 views

Can I always defend using ols, (for example when my dependent variable is ordinal), if I satisfy all CLM assumptions?

Previous reading Let me first say that I went through this post: (How to determine which distribution fits my data best?) and this post: Assumptions of linear models and what to do if the residuals ...
0
votes
1answer
23 views

What are the preliminary analysis before running a logistic regression?

I have a dichotomous variable which represents if a student is accepted or not in a University. In order to do this I have about 60 variables (information of the students: gender, age, etc; their ...
7
votes
1answer
82 views

GAM: Do shrinkage smooth splines also address for concurvity?

I have a gam model with automatic predictor selection based on cubic splines (bs = cr) and SELECT == T or shrinkage cubic splines (bs = cs) and SELECT == F. Now I'm wondering if predictors affected ...
0
votes
0answers
8 views

How do I determine p and q from an acf and pacf plot?

I know you determine p from and q from pacf and acf, but how do you find the optimal number? I still don't quite get how it's done after googling. What is the optimal p and q from the following charts?...
0
votes
0answers
18 views

identify tentative arima model from acf and pacf graphs

here are the ACF % PACF graphs for three different models, kindly please tell teh tentative ARIMA models using these graphs, I will be very much thankful to you
0
votes
0answers
11 views

Are hyperparameters chosen from cross-validation slightly biased towards greater regularization?

I intend to fit a single model to the entire dataset after selecting hyperparameters by k-fold cross-validation. So on each round of training, my model is fit to $\frac{k-1}{k}n$ of my dataset, and ...
1
vote
1answer
41 views

ARIMA (p and q values)

I am trying to do Arima forecasting, i differenced once so d=1, Im not sure what my p and q values need to be, please check screenshots of acf and pacf below:
0
votes
0answers
16 views

How do I know if my data is MA or an AR process?

how do I identify if the process is an ARMA, MA or AR process for example we have:
0
votes
0answers
10 views

Can I use AIC for model selection with same model on data subsets?

I have a class-imbalanced dataset so I divided my data into positive and negative classes (10% pos-90%neg). To model the data I planned to subset the negative data into 8 subsets and then create 8 ...
1
vote
1answer
33 views

Are there any key reasons as to why people would choose between apARCH, gjrGARCH and E-GARCH?

I’ve been doing a lot of R coding with GARCH for my dissertation, I'm coming to the end of my writeup now but have hit a bit of a wall. Obviously, gjrGARCH, apARCH and E-GARCH all allow for asymmetric ...
0
votes
0answers
7 views

Inference making after model selection

I have a data-set (N=1000) with a dependent variable (continuous) and several categorical variable (and one numerical variable). I want to know if the approach that I'm taking is correct when using ...
0
votes
0answers
12 views

joint two-stage estimation with ML

I'm working with data of around 15 variables and half a million observations. To avoid selection bias, I'm trying to incorporate a two-stage joint estimation. I've seen this performed as a censored ...
0
votes
0answers
18 views

Is validation performance sufficient for hyperparameters choice on a small dataset for images multi-classificaiton problem

Problem: multiclass (3-6 classes) images classification (DeepLearning). Dataset size <2000 samples. One class is rare <50 samples. We've conducted several sets of stratified cross-validations ...
0
votes
0answers
25 views

Identify ARIMA Model from ACF & PACF plots

I have plotted ACF & PACF plots for my data, but getting no significance on 1st lag and on further lags it is showing negative lag. please help to identify order of ARIMA model. Thank you
0
votes
2answers
44 views

How to select the best regression model from three models based on hypothesis testing

the following regression models were developed based on the same dataset: model 1: y=a1x1 + b1x2 + c1 model 2: y=a2x1 + b2x3 + c2 model 3: y=a3x1 + b3x4 + c3 where a, b, and c are the regression ...
0
votes
0answers
18 views

Picking a suitable performance metric when comparing the same model but using different sets of training data (Causal inference model)

I am comparing the same models prediction accuracy (Causal Impact) using different control variables as predictors and looking for a metric to decide which set of controls to use. Reading into AIC and ...
0
votes
0answers
4 views

How do I use a baseline of fitness data to score a competition between individuals who do not have similar abilities

Over 10 days the participants exercised as much as they could to get a baseline for each individual's capabilities. They were given a score based on time spent with their heart rate above X. I want ...
1
vote
0answers
7 views

A VECM in logs or a ARDL in the first difference of logs?

Suppose I have a number of time series that appear to have exponential growth at similar rates, with errors I believe to be generally proportional to the level of the variable. I believe that one of ...
0
votes
0answers
54 views

Model smoothness selection for GAMs: GCV vs. REML vs. ML?

I am studying patterns of bird abundances with certain habitat variables and how they vary over time. I am interested in using GAMs with smooth terms for some of the variables. I am, however, confused ...
0
votes
1answer
15 views

Kfold model selection with high train test score deviance

For model hyper parameter tuning, standard k-fold cross validation is being performed. Scoring is done using coefficient of determination, with higher test scores preferred. Dataset size is (100K, 30)....
1
vote
0answers
24 views

BIC and RMSE are Contradict each other for ARIMA Model Selection using R: Do I Err in Theory or in Practice?

I was surprised to see that RMSE and BIC have contradictory trends for the same time-series data. EDITED The procedures in my code are: simulate a 15 AR series of ...
1
vote
1answer
49 views

Singular fit for model and want to retain random intercept

I've read many posts about singular fit issues and this: https://bbolker.github.io/mixedmodels-misc/glmmFAQ.html#singular-models-random-effect-variances-estimated-as-zero-or-correlations-estimated-as--...
0
votes
0answers
24 views

Choosing best regression model

I have a number of regression models from which I'm trying to choose the best performing one. I have computed SSE, AIC and BIC for all, including distributions of errors from predictions on unseen ...
1
vote
1answer
19 views

Parsimonious model - non convergent model decreases fit when removing random slope with near zero variance

I am trying to use Bates et al (2015) recommendation on reducing a maximal model based on variance explained by the random terms. Importantly, all examples in Bates are models with exclusively ...
0
votes
0answers
15 views

Bootstrap for selection among models when best performance is not considered

I understand the title is a bit confusing so I accept any suggestion about it. I have recently finished reading the book Applied Predictive Modeling by Kuhn and Johnson (you can find more information ...
1
vote
1answer
18 views

Comparison of conditional AIC (cAIC) from two or more LMMs

I have a crime data of 300 local divisions (level 1 units) unevenly distributed across 40 districts (level 2 units). I call it Data1. I decided to use cluster analysis to reclassify the level 1 units ...
1
vote
0answers
9 views

Impact of ACF and PACF plots on p,q,d in ARIMA and interpretation of these plots in Python?

I have these two plots: ACF, PACF and my question is: what is an interpretations of these plots and by looking on this plots what values of parameters p, q ,d should I choose for my ARIMA model ?
1
vote
0answers
16 views

Model Selection for Multivariate Time Series Classification Problem

Problem Definition: You are given a dataset of $N$ different features. Most of the features are actually calculations of their own time series of set length (i.e A 5-time step weighted moving average)...
1
vote
0answers
27 views

What counts as a parameter for AIC?

I know this question has been asked before (e.g. here Meaning of 'number of parameters' in AIC), but I am still confused. What exactly makes something as a parameter for the AIC penalty, ...
1
vote
0answers
23 views

Is the model with the lowest AIC value always the preferred one? [duplicate]

I have a question regarding model selection based on the AIC criterion. I have 5 predictor variables that I include in my models (no interactions are included) and create all possible model ...
2
votes
0answers
41 views

Interview question: train/test error and “best” model

I recently had a puzzling interview question and I am wondering whether anybody can tell me the intended answer. The question shows train and test error for three models plotted against the number of ...
0
votes
0answers
11 views

Adaptive cutpoint selection on ROC curves based on changing environments

I have built a classic binary classifier and constructed a ROC curve for it, like the following: In this case, the positive class represents "bad" things that should be excluded. In the ...
1
vote
0answers
26 views

ARIMA PACF plot interpretation

I have this dataset about voters in a country There is a clear trend therefore I differentiate the data once to achieve a stationary data The resulting acf and pacf plots are below Based on my ...
1
vote
0answers
20 views

Do conventional thresholds for global fit indices (e.g. AIC) hold for models based on very large data sets?

Problem/Question in short: I have estimated 5 generalized linear mixed models and subsequently compared their levels of relative fit according to AIC. These models are based on a very large dataset of ...
1
vote
1answer
24 views

Multiple comparisons correction, for alpha-less criteria like AIC

When performing multiple hypothesis tests, for example in stepwise model selection, we need to apply something like the Bonferroni correction to the alpha/significance value in order to avoid too many ...
0
votes
0answers
18 views

Determining p and q parameters of ARIMA from ACF and PACF plots

I have three time series that I have differenced with the order of 1 to achieve stationarity. Based on these ACF and PACF plots, what would be the values of p and q in the ARIMA model for each time ...
0
votes
0answers
17 views

fit suitable AR-GARCH models to returns in MATLAB

I fit AR(1)-GARCH(1,1) models to 100 shares return, but at almost all of them the fitted AR(1) model is not stationary (AR{1}=0.99). How I can find a suitable same model? (for example by using the AIC ...
0
votes
1answer
23 views

How can we select the best GARCH model by carrying out likelihood ratio test?

I have carried out the likelihood ratios of different GARCH models. GARCH(1,1) and GARCH(1,0)- Rejected null hypothesis so I chose GARCH(1,1) to do more sophistication. GARCH(3,1) and GARCH(1,1)- ...
1
vote
1answer
18 views

Which choice model to analyse my binary stated choice experiment data to estimate willingess-to-pay?

I followed the recipe of a stated-choice experiment in political science https://doi.org/10.1093/pan/mpt024 (they call it "conjoint" but I think this term is debated). In the end I made the ...
1
vote
1answer
28 views

Analyzing animal behavioural data in R Studio (GLMM) and struggling to understand how to set up the model

I'm trying to determine whether there was any differences in behaviour (count data, frequency per 30 minute observations, categorized into 5 categories [e.g., affiliative, agonistic, etc.]) of a ...
0
votes
1answer
27 views

Do you need a separate test set when performing nested cross-validation?

I understand that the inner cross-validation error can be biased and overly optimistic so that's why people use nested cross-validation to get unbiased generalization error. But we also have a hold ...
0
votes
1answer
17 views

cs231n: train/val accuracy and number of parameters

In this cs231 lection note there is a counterintuitive quote: The gap between the training and validation accuracy indicates the amount of overfitting. ... The other possible case is when the ...
0
votes
0answers
17 views

Regression analysis with binary independent (categorical) variables and continous dependent variable

I am working on a dataset with 14 binary (0 or 1) independent variables (product features) and trying to measure a continous dependent variable (product price). I tried doing a regression analysis to ...
3
votes
1answer
46 views

Drawing the Line between a Model and a parameters in Bayesian Model selection

In Bayesian Model Selection, we first compute the evidence: $$ p(D|m) = \int_{\theta} p(D|\theta) p(\theta|m) d\theta $$ Then, we select the model that maximizes, which is a MAP estimator of the model....
0
votes
0answers
12 views

Help with identifying regression model for sequences

Let's say I have a regression task, but features are sequences of "letters", so that the order is important. The set of all possible letters is relatively small. Example: ...
0
votes
0answers
15 views

Truncated NB? Or go with linear?

I am not here to ask about which R package to use, etc. I am here about model selection. The question I am trying to analyze is - What is the association between Indicator 1 and age, sex, and group? ...
2
votes
0answers
35 views

Comparing fits of linear models that use different data spans of a curve (and why does AIC seem to work)

Several related questions have been asked. This one is similar, but it does not match this question exactly. Also, i seem to have results that contradict the accepted answer there. Data An imperfect ...

1
2 3 4 5
35