# Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

1,758 questions
Filter by
Sorted by
Tagged with
432 views

### RMSE or MAPE? which one to choose for accuracy?

I have a weekly times series for which I would like to find the best fit model. So far I've tried arima, Harmonic regression with arima error, neural network and in the end I would like to decide ...
51 views
+50

### Different train/test for model selection

Imagine that I have a function that is somewhat like the following: ...
17 views

### SARIMA: Determining the integration number

i am working with a time series that after calculating the first difference, it remains non-stationary. When plotting the series I see there is some seasonality at half and end of year. I would like ...
36 views

### Selecting a betareg model out of all possible combinations of the EVs

I'm working on a dataset with response variable in [0,1] and n=61 observations, and trying to fit a model with betareg(). After posting my former question, I ...
31 views

### Can TBATS be used in the case of deterministic seasonality?

I have a monthly dataset which consist of 176 points. I validated that it is stationary by adf.test and it has deterministic seasonality by ...
24 views

### Best-Subset Regression based on BIC versus Forward Selection based on AIC

I am trying to get a better grasp of BIC and AIC scores. I know BIC has a harsher penalty than AIC regarding model size (it prefers smaller, less complex models). Suppose there is a situation where I ...
47 views

### How to select P and Q in ARIMA

I am new to using ARIMA and I would like to know how to determine the p, q values of ARIMA by PCF and PACF. Here is the raw data figure. The raw data is a human glucose data collected.The blue data is ...
41 views

### Identification Problem of SARIMA

I am trying to make time series analysis with SARIMA and I have a question. My dataset is a seasonal dataset. I validated that I have stationary series by KPSS test. I also found the following ...
21 views

### 2 groups - some participants are only in one group, some in both: how to analyze?

I want to compare questionnaire scores about symptom X arising from trigger Y vs. trigger Z. Usually I'd go for a between-subject design. However, some participants (~50% of the sample) have had X as ...
9 views

### Assessing model fit with predicted vs actual values with error in both?

I have a non-linear model that predicts the number of moose each year from three parameters. These parameters each have a measured error which I propagated in my prediction/estimate of moose number ...
45 views

### Sales forecasting. What model to choose and how to interpret ACF/PACF?

I have the following raw sales timeseries: Clearly it's not stationary since it does have somewhat of an increasing trend (non-stationarity also verified by DF-test). By differencing I obtain the ...
51 views

### How does repeated k-fold cross validation identify model instability?

In these threads 1,2,3, cbeleites mentions that in a single k-fold cross validation you cannot tell whether the variance is caused by model instability or using a different test set. Hence, one can ...
151 views

### R best subset selection to find (1) what the single best variable is, (2) what the first variable is, and (3) what the last variable is

I am trying to analyse this dataset using best subset selection: I perform best subset selection using the R regsubsets function as follows: ...
22 views

### Highly important categorical variable with many values and only few data points per value

Let's say I've got a dataset of music albums. As predictors, I have the artist, the genre, the year it was made plus several others (categorical and numeric). I want to predict the number of copies ...
102 views

### How to get R-Squared after doing stepwise model selection in regression in R

I am using R commander to do stepwise model selection in a linear model. When I run stepwise model selection, it reduces some variables, and finally, a model with AIC is provided. However, it does not ...
19 views

### Is there a universal way to calculate model likelihoods for an arbitrary distribution?

I have no background in statistics but have been tasked to use AIC and BIC to select a model for some observed experimental data. The population data cannot be assumed to obey any particular ...
42 views

### Which if these two models works better?

I have this time series I want to perform polynomial regression on, to estimate the trend. To start, I tried using only a second order polynomial, these are the results (AIC=30.37105) We can see how ...
38 views

### How to interpret increase to AIC and adjusted r-squared?

I understand adjusted r-squared and AIC can be used to select an ideal model from a group. Higher AIC is worse but higher ar2 is better. After adding a categorical variable to an OLS model, my ...
83 views

### Using Silhouette Score to evaluate different clustering algorithm

I am trying to compare different clustering algorithms on a dataset and compare the model performance. Since the dataset is quite big (56 features), I applied PCA to reduce the number of features to ...
16 views

### Is there some AIC/BIC analog of significance measures by coefficient?

Is there some AIC/BIC analog of significance measures by coefficient? Are there numbers that are the AIC/BIC counterparts of significance tests? I know that you can take the AIC or BIC of a model, ...
41 views

### AIC/BIC Based Model Selection And Sample Size

I am using BIC to tune a lasso estimation and select the features that will be used in further analysis. The data is quite large, and I have some prior domain knowledge on it, so I split it by several ...
52 views

### Selecting p,q,d for ARIMA and overfitting. Shouldn't the parameters be tuned on a training set?

I have seen multiple tutorials [example link] for ARIMA where they select the p,q,d parameters for it based on the whole time series. Then, after deciding on the model parameters they want to use, ...
26 views

### Can you use AIC scores with mixed-effects models?

Is it possible to do model selection with AIC scores having linear mixed-effects models? How about binary logistic mixed-effects? Moreover, how should models be specified? e.g., if we have a model ...
25 views

### Can the sufficient statistics simplify Bayesian model selection?

When selecting a model within the Bayesian framework we can choose the model having highest probability given the data. This is proportional to the marginal likelihood or evidence for the data when we ...
32 views

### AR and MA terms for a highly irregular time-series?

I have been fussing over the ACF and PACF plots with 1 order of differencing (passed the ADF test) but I haven't found any information that deals with ACF and PACF plots that have lags as large as ...
23 views

### Selecting ARIMA model based on PACF and ACF graphs

I am facing two pairs of ACF and PACF graphs. Unfortunately, I cannot use auto.arima, but I need to make sure my intuition is correct. 1. 2. 1. I believe that an ARIMA(1,1,0) would produce similar ...
56 views

### cross-validation and test set

Apologies if this has been answered before elsewhere. Answers I have read so far have only confused me further. Essentially, I want to check whether I can use the test set to choose betweeen two ...
15 views

### How to determine the seasonal component order of an ARIMA model?

I am implementing two ARIMA models in R, I used a first order differencing and a seasonal differencing to get the stationary data, like this: For the first one I used auto.arima() and got an ARIMA(...
34 views

### Single or multiple model. How to know beforehand?

I am building a probability of default model based on behavioral information. The dataset is a loan portfolio, which contains 4 types of loans: mortgage, unsecured loans, car loans and credit cards. ...
23 views

### Dimension selection based on test accuracy

consider that I have a dataset with train, validation and test set and I want to train the pipeline PCA+logistic regression classifier. So far, for a specific k (that is the reduced data dimension ...
25 views

### Why does the best fit model (lower AIC) yield higher p values than models with higher AIC? [closed]

Background: I am running a model selection in R that includes 1, 2, and 3-covariate models. Each model aims to determine the effect of environmental covariates in the occupancy of different species in ...
27 views

### Model/variable selection in binomial models for marble counts?

Thank you in advance for your help with this interesting question that has come up in my research. I changed the problem to bags/marbles to simplify. Problem description There are N bags, each ...
32 views

### Time Series Forecasting Process With Regard to Training and Test Sets

I'm a bit confused about the process order in doing proper time series analysis/forecasting. Is it: Stationary/seasonality checks, do any transformations required Candidate model selection using ACF, ...
991 views

### What if there is no true data-generating process?

I've been engaging in a number of forecasting efforts recently, and have rediscovered a well-known truth: That combinations of different forecasts are generally better than the forecasts themselves. ...
45 views

### Should one use the usual splitting (Learning/Validation/Test) when using cross-validation?

Say you want to tune several parameters of your model using $N$ data. What you usually do is splitting your $N$ data into 3 sets: learning set: used to build your model; validation set: used to ...
37 views

### Why do we choose the hyperparameters that gives the lowest validation error? Do we assume that it also gives the lowest generalization error?

The usual way of selecting hyperparameters is to tune it on the validation set and select the hyperparameters that gives the lowest validation error (Lets assume the validation sample is large so we ...
19 views

### Determining Order of ARMA model

I'm working with time series data in R and I'm trying to figure out the order of an ARMA model. I'm not quite sure how to interpret the PACF and ACF plots. The series was proven stationary through an ...
28 views

### VAR model selection: is AIC an appropriate measure given that the sample size changes depending on the number of lags?

Given that the sample size of a VAR (or a similar model: VARX, SVAR etc.) reduces by $1$ for each extra dependent-variable lag that I introduce (since we need to drop the empty rows, or ...
51 views

### Right kind of statistical analysis?

I'm performing a user study where we are attempting to compare two forms of measurement of participant state-of-mind. Both measurements are numerical scores - one is self-reported, and the other is ...
37 views

### Model complexity vs regularization

How do the complexity of a model and regularization behave with each other? Like we could decrease the degree of a polynomial or add a regularization term. Or both? In other words: Why is there ...
14 views

### Are tree based models for regression/classification 'endlessly' trainable?

I know this is a relatively simple question that I could answer if I understood trees more. An ANN can be trained indefinitely, especially if it is deep. What other models besides networks have this ...
35 views

### VAR - VARX model selection

Suppose I have three stationary economic time series $y_i$ that are not cointegrated and I want to investigate the relationship between them. I happen to be unsure about the "endogenousness" ...
42 views

### Is it wrong to compare multiple models on the same test set and choose the best model?

Suppose we split a dataset into 3 parts (train, validation, and test). I know that it's important to make sure the test set doesn't influence our decisions during model selection or hyperparameter ...
16 views

### Lag choice for a VAR model

There is something confusing (for me) about this question. I'm working on a VAR model (7 time series) where i've checked for Granger causality (yes) and stationarity (yes). Now, according to the AIC ...
46 views

### ARMA zero coefficients (not statistically significant)

I seem to have forgotten what to do when the coefficients of the time series model are not statistically significant like $\phi_2$, $\phi_3$, $\phi_4$ below. A university course taught us to set each ...
50 views

### automatic model selection of linear models

https://i.stack.imgur.com/uYayd.gif https://i.stack.imgur.com/bAEMc.gif tail -c +43 uYayd.gif > TROW.tsv tail -c +43 bAEMc.gif > AABB.tsv Using the two ...
77 views

### How can we assume the models are exhaustive in Bayesian Model Averaging?

Bayesian model averaging is justified using the law of total probability which requires the the set of models that we average over to be exhaustive. Shouldn’t we prove that the set of models are ...
52 views

### Interpretation of Elastic net having too low or high value of alpha

Often I found the situation that the elastic model what I fitted has optimal alpha value at 0 or 1. Or not only that situation, but also there some alphas go near to 0 or 1.(ex. 0.1 or 0.9) My ...