Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

Filter by
Sorted by
Tagged with
0 votes
1 answer
209 views

Selecting ARIMA model based on PACF and ACF graphs

I am facing two pairs of ACF and PACF graphs. Unfortunately, I cannot use auto.arima, but I need to make sure my intuition is correct. 1. 2. 1. I believe that an ARIMA(1,1,0) would produce similar ...
user avatar
0 votes
1 answer
105 views

cross-validation and test set

Apologies if this has been answered before elsewhere. Answers I have read so far have only confused me further. Essentially, I want to check whether I can use the test set to choose betweeen two ...
user avatar
1 vote
0 answers
54 views

How to determine the seasonal component order of an ARIMA model?

I am implementing two ARIMA models in R, I used a first order differencing and a seasonal differencing to get the stationary data, like this: For the first one I used auto.arima() and got an ARIMA(...
user avatar
2 votes
1 answer
41 views

Single or multiple model. How to know beforehand?

I am building a probability of default model based on behavioral information. The dataset is a loan portfolio, which contains 4 types of loans: mortgage, unsecured loans, car loans and credit cards. ...
user avatar
0 votes
1 answer
25 views

Dimension selection based on test accuracy

consider that I have a dataset with train, validation and test set and I want to train the pipeline PCA+logistic regression classifier. So far, for a specific k (that is the reduced data dimension ...
user avatar
1 vote
0 answers
32 views

Why does the best fit model (lower AIC) yield higher p values than models with higher AIC? [closed]

Background: I am running a model selection in R that includes 1, 2, and 3-covariate models. Each model aims to determine the effect of environmental covariates in the occupancy of different species in ...
user avatar
0 votes
0 answers
37 views

Model/variable selection in binomial models for marble counts?

Thank you in advance for your help with this interesting question that has come up in my research. I changed the problem to bags/marbles to simplify. Problem description There are N bags, each ...
user avatar
2 votes
1 answer
105 views

Time Series Forecasting Process With Regard to Training and Test Sets

I'm a bit confused about the process order in doing proper time series analysis/forecasting. Is it: Stationary/seasonality checks, do any transformations required Candidate model selection using ACF, ...
user avatar
15 votes
2 answers
1k views

What if there is no true data-generating process?

I've been engaging in a number of forecasting efforts recently, and have rediscovered a well-known truth: That combinations of different forecasts are generally better than the forecasts themselves. ...
user avatar
  • 2,617
1 vote
1 answer
49 views

Should one use the usual splitting (Learning/Validation/Test) when using cross-validation?

Say you want to tune several parameters of your model using $N$ data. What you usually do is splitting your $N$ data into 3 sets: learning set: used to build your model; validation set: used to ...
user avatar
  • 161
3 votes
1 answer
72 views

Why do we choose the hyperparameters that gives the lowest validation error? Do we assume that it also gives the lowest generalization error?

The usual way of selecting hyperparameters is to tune it on the validation set and select the hyperparameters that gives the lowest validation error (Lets assume the validation sample is large so we ...
user avatar
  • 179
0 votes
0 answers
34 views

Determining Order of ARMA model

I'm working with time series data in R and I'm trying to figure out the order of an ARMA model. I'm not quite sure how to interpret the PACF and ACF plots. The series was proven stationary through an ...
user avatar
1 vote
1 answer
208 views

VAR model selection: is AIC an appropriate measure given that the sample size changes depending on the number of lags?

Given that the sample size of a VAR (or a similar model: VARX, SVAR etc.) reduces by $1$ for each extra dependent-variable lag that I introduce (since we need to drop the empty rows, or ...
user avatar
  • 252
0 votes
1 answer
53 views

Right kind of statistical analysis?

I'm performing a user study where we are attempting to compare two forms of measurement of participant state-of-mind. Both measurements are numerical scores - one is self-reported, and the other is ...
user avatar
3 votes
0 answers
45 views

Model complexity vs regularization

How do the complexity of a model and regularization behave with each other? Like we could decrease the degree of a polynomial or add a regularization term. Or both? In other words: Why is there ...
user avatar
  • 2,309
0 votes
0 answers
14 views

Are tree based models for regression/classification 'endlessly' trainable?

I know this is a relatively simple question that I could answer if I understood trees more. An ANN can be trained indefinitely, especially if it is deep. What other models besides networks have this ...
user avatar
1 vote
1 answer
72 views

VAR - VARX model selection

Suppose I have three stationary economic time series $y_i$ that are not cointegrated and I want to investigate the relationship between them. I happen to be unsure about the "endogenousness" ...
user avatar
  • 252
7 votes
1 answer
812 views

Is it wrong to compare multiple models on the same test set and choose the best model?

Suppose we split a dataset into 3 parts (train, validation, and test). I know that it's important to make sure the test set doesn't influence our decisions during model selection or hyperparameter ...
user avatar
0 votes
0 answers
34 views

Lag choice for a VAR model

There is something confusing (for me) about this question. I'm working on a VAR model (7 time series) where i've checked for Granger causality (yes) and stationarity (yes). Now, according to the AIC ...
user avatar
  • 1
1 vote
0 answers
88 views

ARMA zero coefficients (not statistically significant)

I seem to have forgotten what to do when the coefficients of the time series model are not statistically significant like $\phi_2$, $\phi_3$, $\phi_4$ below. A university course taught us to set each ...
user avatar
  • 441
1 vote
1 answer
54 views

automatic model selection of linear models

https://i.stack.imgur.com/uYayd.gif https://i.stack.imgur.com/bAEMc.gif tail -c +43 uYayd.gif > TROW.tsv tail -c +43 bAEMc.gif > AABB.tsv Using the two ...
user avatar
1 vote
3 answers
92 views

How can we assume the models are exhaustive in Bayesian Model Averaging?

Bayesian model averaging is justified using the law of total probability which requires the the set of models that we average over to be exhaustive. Shouldn’t we prove that the set of models are ...
user avatar
0 votes
0 answers
132 views

Interpretation of Elastic net having too low or high value of alpha

Often I found the situation that the elastic model what I fitted has optimal alpha value at 0 or 1. Or not only that situation, but also there some alphas go near to 0 or 1.(ex. 0.1 or 0.9) My ...
user avatar
1 vote
4 answers
604 views

Thousands of features and only 70 samples

I am working on a regression problem where I have around 5-10 thousand features and have only 65 samples. I am training my algorithm on 55 samples and testing on 10 samples. I am using both Pearson ...
user avatar
  • 11
1 vote
1 answer
375 views

Classical difference-in-differences: Coding the time (post) variable when treatment starts at different times

I have panel data with 40 treated cases and 40 control cases. I thought about the application of the 'classical' difference-in-differences (DiD) equation with the following linear regression model: $y ...
user avatar
0 votes
2 answers
48 views

How to choose between 2 strategies to train a Deep Learning model on an unbalanced Dataset?

I have a Training Set of respiratory disease sounds, so there are 2 classes: 0 for respiratory sounds of healthy patients. 1 for breathing sounds of patients with a disease. The Training Set is ...
user avatar
0 votes
0 answers
152 views

Variable selection in Bayesian hierarchical models with R-INLA

I'm working with Bayesian hierarchichal regressions fitted with R-INLA. I would like to simplify my model by reducing the number of covariates. According to my understanding, Bayesian variable ...
user avatar
2 votes
1 answer
863 views

Setting random seed for final neural network model

I'm a bit uncertain about the correct experimental procedure on when to set random seeds when training machine learning models with random components or initializations. Let's say I had to create a ...
user avatar
0 votes
1 answer
40 views

ets() function does not minimize AICc?

I have a question that is similar to this question: ETS function in forecast package is not choosing minimized AICc I see what the author of that question misunderstood but I basically have a reverse ...
user avatar
  • 1
1 vote
0 answers
231 views

Good fit, but poor residuals in GAM

I am very new to GAM. Data represent: Weight loss ~ Days after treatment measured in two groups: A (black dots) and B (red dots). I magage to get an excellent fit, and yet I get what looks like a poor ...
user avatar
  • 111
1 vote
1 answer
115 views

MLE for choosing models

let say I have 2 models with the same number of parameters. I implemented the Gaussian Log-Likelihood in R, based on real data for both models. In the end, one model provides a higher MLE than the ...
user avatar
  • 11
0 votes
0 answers
18 views

AICc within SOCPROG 2.9

My question is operational: how can I set AIC correction for small samples in SOCPROG 2.9? I would estimate movement into and out of the study area, and residency patterns of marked individuals from a ...
user avatar
0 votes
1 answer
42 views

Why is the AIC score doubled and does it matter?

I have used AIC to do model selection before by just following the classic formula: AIC=2k-2L But as far as I understand the absolute value of this score doesn't matter, only the relative score ...
user avatar
  • 103
2 votes
1 answer
95 views

Cost of Nested Cross-Validation

What is the cost of nested cross-validation in terms of the number of times the algorithm needs to perform a fit-evaluate step? Based on this description of the algorithm, I think the answer is: $n \...
user avatar
0 votes
1 answer
283 views

Neural network using Orange

I'm using Orange Data Mining in a regression analysis applying Artificial Neural Network (ANN). Some works suggest defining the number of neurons in the input layer as the number of variables. The ...
user avatar
0 votes
0 answers
44 views

Model fitting VS model selection: what works best?

Suppose we have two candidate models to predict a variable $y$ given a variable $x$, where $\alpha$ is a model parameter. $$\hat y=M_1(x,\alpha)$$ $$\hat y=M_2(x,\alpha)$$ Conceptually, we could ...
user avatar
  • 1,046
2 votes
0 answers
236 views

R Squared (OOB) and R Square from correlation of prediction of test set is different?

I'm using simulated data and fitting Random Forest model for regression on a training dataset. What is confusing me is that after running Random Forest, I got R Squared (OOB) equal to 0.14. But when I ...
user avatar
0 votes
0 answers
188 views

Linear model selection (AIC/dredge function) with non parametric data (residuals are autocorrelated)

I used the dredge function with MuMIn & lme4 package to do a linear model selection with AIC principle. I have about 10 ...
user avatar
2 votes
1 answer
117 views

Is BIC asymptotically efficient for minimizing prediction error if the true model is being considered?

If a set of models is being compared using BIC and AIC, given the fact that the true model (the one which generated the data) is in this set (and given the other assumptions that guarantee BIC ...
user avatar
3 votes
1 answer
112 views

Fixed effects in DAGs

Let's imagine I'm interested in studying the causal effect of beliefs in some ideas and behavior related to these ideas (say, if I believe sunscreen is good for my health, I use more sunscreen etc.). ...
user avatar
0 votes
0 answers
25 views

Interprete AIC as significance test and determine significance level for nested models

Let's say we have two nested models. The smaller one (corresponds to $H_0$) has $p_0$ parameters, so its AIC is given by $$AIC_0=-2\log L_0+2p_0$$ The larger model has $p_1:=p_0+d$ parameters of which ...
user avatar
0 votes
0 answers
20 views

How can compare suggestion models with different performances?

I have 4 class binary classification models. That models identify which class a particular students is suitable for. For example, we have user 1 and 4 classes ...
user avatar
  • 101
1 vote
0 answers
70 views

Split one model into two models for a classification problem

In the classification problem, I am working on right now, I have to classify transactions mainly with text data. The classes of the training set can easily be divided into class sets with a negative ...
user avatar
  • 11
0 votes
0 answers
41 views

Stabilizing nested cross-validation

Quite recently I stumbled upon several posts here on nested cross-validation, which showed me how wrong my understanding was of such procedure. Now, trying to put all the pieces together, I still have ...
user avatar
  • 1
3 votes
1 answer
48 views

Can I use AIC for path models and non-path models?

I am testing competing hypotheses where one hypothesis contains a mediation effects that can be modeled using a path model. The other hypothesis does not include a mediation effect and therefore can ...
user avatar
3 votes
2 answers
349 views

MultiClass Classification - Training OvO and OvA

I like to know how OvO (One vs One) and OvA (One vs All) models are trained in multiclass classification problem. To keep it simple, we have 4 classes, each of which has 1000 datapoints. What are the ...
user avatar
0 votes
1 answer
40 views

ANOVA for comparing same linear models

I have trained two linear regressor models with the same response $Y$ and the same features $x_1$ and $x_2$, they are basically the same model, however only the training data differs: in model1 data ...
user avatar
0 votes
0 answers
313 views

How to determine the optimal lag order of asymmetric innovation for a GARCH?

In the python package arch_model, there is an option with which we can give the "lag order of the asymmetric innovation" when we estimate GARCH model. Are there any ways to find the optimal ...
user avatar
  • 539
19 votes
4 answers
2k views

Why does the Akaike Information Criterion (AIC) sometimes favor an overfitted model?

As an exercise to develop practical experience working with model selection criteria, I computed fits of the highway mpg vs. engine displacement data from the tidyverse mpg example data set using ...
user avatar
  • 3,002
0 votes
1 answer
85 views

Reducing a linear regression (OLS) model by dropping non-significant coefficients

Would it be proper for me to reduce a model by iterating though the coefficients and dropping the ones with high p-values and then refitting and doing this again until all coefficients are significant?...
user avatar
  • 441

1 2 3
4
5
38