Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

Filter by
Sorted by
Tagged with
3
votes
2answers
432 views

RMSE or MAPE? which one to choose for accuracy?

I have a weekly times series for which I would like to find the best fit model. So far I've tried arima, Harmonic regression with arima error, neural network and in the end I would like to decide ...
1
vote
2answers
51 views
+50

Different train/test for model selection

Imagine that I have a function that is somewhat like the following: ...
0
votes
0answers
17 views

SARIMA: Determining the integration number

i am working with a time series that after calculating the first difference, it remains non-stationary. When plotting the series I see there is some seasonality at half and end of year. I would like ...
0
votes
0answers
36 views

Selecting a betareg model out of all possible combinations of the EVs

I'm working on a dataset with response variable in [0,1] and n=61 observations, and trying to fit a model with betareg(). After posting my former question, I ...
1
vote
0answers
31 views

Can TBATS be used in the case of deterministic seasonality?

I have a monthly dataset which consist of 176 points. I validated that it is stationary by adf.test and it has deterministic seasonality by ...
0
votes
1answer
24 views

Best-Subset Regression based on BIC versus Forward Selection based on AIC

I am trying to get a better grasp of BIC and AIC scores. I know BIC has a harsher penalty than AIC regarding model size (it prefers smaller, less complex models). Suppose there is a situation where I ...
0
votes
2answers
47 views

How to select P and Q in ARIMA

I am new to using ARIMA and I would like to know how to determine the p, q values of ARIMA by PCF and PACF. Here is the raw data figure. The raw data is a human glucose data collected.The blue data is ...
0
votes
1answer
41 views

Identification Problem of SARIMA

I am trying to make time series analysis with SARIMA and I have a question. My dataset is a seasonal dataset. I validated that I have stationary series by KPSS test. I also found the following ...
0
votes
0answers
21 views

2 groups - some participants are only in one group, some in both: how to analyze?

I want to compare questionnaire scores about symptom X arising from trigger Y vs. trigger Z. Usually I'd go for a between-subject design. However, some participants (~50% of the sample) have had X as ...
0
votes
0answers
9 views

Assessing model fit with predicted vs actual values with error in both?

I have a non-linear model that predicts the number of moose each year from three parameters. These parameters each have a measured error which I propagated in my prediction/estimate of moose number ...
0
votes
1answer
45 views

Sales forecasting. What model to choose and how to interpret ACF/PACF?

I have the following raw sales timeseries: Clearly it's not stationary since it does have somewhat of an increasing trend (non-stationarity also verified by DF-test). By differencing I obtain the ...
2
votes
2answers
51 views

How does repeated k-fold cross validation identify model instability?

In these threads 1,2,3, cbeleites mentions that in a single k-fold cross validation you cannot tell whether the variance is caused by model instability or using a different test set. Hence, one can ...
1
vote
0answers
151 views

R best subset selection to find (1) what the single best variable is, (2) what the first variable is, and (3) what the last variable is

I am trying to analyse this dataset using best subset selection: I perform best subset selection using the R regsubsets function as follows: ...
0
votes
1answer
22 views

Highly important categorical variable with many values and only few data points per value

Let's say I've got a dataset of music albums. As predictors, I have the artist, the genre, the year it was made plus several others (categorical and numeric). I want to predict the number of copies ...
1
vote
2answers
102 views

How to get R-Squared after doing stepwise model selection in regression in R

I am using R commander to do stepwise model selection in a linear model. When I run stepwise model selection, it reduces some variables, and finally, a model with AIC is provided. However, it does not ...
0
votes
0answers
19 views

Is there a universal way to calculate model likelihoods for an arbitrary distribution?

I have no background in statistics but have been tasked to use AIC and BIC to select a model for some observed experimental data. The population data cannot be assumed to obey any particular ...
0
votes
1answer
42 views

Which if these two models works better?

I have this time series I want to perform polynomial regression on, to estimate the trend. To start, I tried using only a second order polynomial, these are the results (AIC=30.37105) We can see how ...
1
vote
0answers
38 views

How to interpret increase to AIC and adjusted r-squared?

I understand adjusted r-squared and AIC can be used to select an ideal model from a group. Higher AIC is worse but higher ar2 is better. After adding a categorical variable to an OLS model, my ...
0
votes
0answers
83 views

Using Silhouette Score to evaluate different clustering algorithm

I am trying to compare different clustering algorithms on a dataset and compare the model performance. Since the dataset is quite big (56 features), I applied PCA to reduce the number of features to ...
0
votes
0answers
16 views

Is there some AIC/BIC analog of significance measures by coefficient?

Is there some AIC/BIC analog of significance measures by coefficient? Are there numbers that are the AIC/BIC counterparts of significance tests? I know that you can take the AIC or BIC of a model, ...
0
votes
0answers
41 views

AIC/BIC Based Model Selection And Sample Size

I am using BIC to tune a lasso estimation and select the features that will be used in further analysis. The data is quite large, and I have some prior domain knowledge on it, so I split it by several ...
3
votes
1answer
52 views

Selecting p,q,d for ARIMA and overfitting. Shouldn't the parameters be tuned on a training set?

I have seen multiple tutorials [example link] for ARIMA where they select the p,q,d parameters for it based on the whole time series. Then, after deciding on the model parameters they want to use, ...
0
votes
0answers
26 views

Can you use AIC scores with mixed-effects models?

Is it possible to do model selection with AIC scores having linear mixed-effects models? How about binary logistic mixed-effects? Moreover, how should models be specified? e.g., if we have a model ...
2
votes
0answers
25 views

Can the sufficient statistics simplify Bayesian model selection?

When selecting a model within the Bayesian framework we can choose the model having highest probability given the data. This is proportional to the marginal likelihood or evidence for the data when we ...
1
vote
1answer
32 views

AR and MA terms for a highly irregular time-series?

I have been fussing over the ACF and PACF plots with 1 order of differencing (passed the ADF test) but I haven't found any information that deals with ACF and PACF plots that have lags as large as ...
0
votes
1answer
23 views

Selecting ARIMA model based on PACF and ACF graphs

I am facing two pairs of ACF and PACF graphs. Unfortunately, I cannot use auto.arima, but I need to make sure my intuition is correct. 1. 2. 1. I believe that an ARIMA(1,1,0) would produce similar ...
0
votes
1answer
56 views

cross-validation and test set

Apologies if this has been answered before elsewhere. Answers I have read so far have only confused me further. Essentially, I want to check whether I can use the test set to choose betweeen two ...
1
vote
0answers
15 views

How to determine the seasonal component order of an ARIMA model?

I am implementing two ARIMA models in R, I used a first order differencing and a seasonal differencing to get the stationary data, like this: For the first one I used auto.arima() and got an ARIMA(...
2
votes
1answer
34 views

Single or multiple model. How to know beforehand?

I am building a probability of default model based on behavioral information. The dataset is a loan portfolio, which contains 4 types of loans: mortgage, unsecured loans, car loans and credit cards. ...
0
votes
1answer
23 views

Dimension selection based on test accuracy

consider that I have a dataset with train, validation and test set and I want to train the pipeline PCA+logistic regression classifier. So far, for a specific k (that is the reduced data dimension ...
1
vote
0answers
25 views

Why does the best fit model (lower AIC) yield higher p values than models with higher AIC? [closed]

Background: I am running a model selection in R that includes 1, 2, and 3-covariate models. Each model aims to determine the effect of environmental covariates in the occupancy of different species in ...
0
votes
0answers
27 views

Model/variable selection in binomial models for marble counts?

Thank you in advance for your help with this interesting question that has come up in my research. I changed the problem to bags/marbles to simplify. Problem description There are N bags, each ...
2
votes
1answer
32 views

Time Series Forecasting Process With Regard to Training and Test Sets

I'm a bit confused about the process order in doing proper time series analysis/forecasting. Is it: Stationary/seasonality checks, do any transformations required Candidate model selection using ACF, ...
13
votes
2answers
991 views

What if there is no true data-generating process?

I've been engaging in a number of forecasting efforts recently, and have rediscovered a well-known truth: That combinations of different forecasts are generally better than the forecasts themselves. ...
1
vote
1answer
45 views

Should one use the usual splitting (Learning/Validation/Test) when using cross-validation?

Say you want to tune several parameters of your model using $N$ data. What you usually do is splitting your $N$ data into 3 sets: learning set: used to build your model; validation set: used to ...
3
votes
1answer
37 views

Why do we choose the hyperparameters that gives the lowest validation error? Do we assume that it also gives the lowest generalization error?

The usual way of selecting hyperparameters is to tune it on the validation set and select the hyperparameters that gives the lowest validation error (Lets assume the validation sample is large so we ...
0
votes
0answers
19 views

Determining Order of ARMA model

I'm working with time series data in R and I'm trying to figure out the order of an ARMA model. I'm not quite sure how to interpret the PACF and ACF plots. The series was proven stationary through an ...
1
vote
1answer
28 views

VAR model selection: is AIC an appropriate measure given that the sample size changes depending on the number of lags?

Given that the sample size of a VAR (or a similar model: VARX, SVAR etc.) reduces by $1$ for each extra dependent-variable lag that I introduce (since we need to drop the empty rows, or ...
0
votes
1answer
51 views

Right kind of statistical analysis?

I'm performing a user study where we are attempting to compare two forms of measurement of participant state-of-mind. Both measurements are numerical scores - one is self-reported, and the other is ...
3
votes
0answers
37 views

Model complexity vs regularization

How do the complexity of a model and regularization behave with each other? Like we could decrease the degree of a polynomial or add a regularization term. Or both? In other words: Why is there ...
0
votes
0answers
14 views

Are tree based models for regression/classification 'endlessly' trainable?

I know this is a relatively simple question that I could answer if I understood trees more. An ANN can be trained indefinitely, especially if it is deep. What other models besides networks have this ...
1
vote
1answer
35 views

VAR - VARX model selection

Suppose I have three stationary economic time series $y_i$ that are not cointegrated and I want to investigate the relationship between them. I happen to be unsure about the "endogenousness" ...
5
votes
0answers
42 views

Is it wrong to compare multiple models on the same test set and choose the best model?

Suppose we split a dataset into 3 parts (train, validation, and test). I know that it's important to make sure the test set doesn't influence our decisions during model selection or hyperparameter ...
0
votes
0answers
16 views

Lag choice for a VAR model

There is something confusing (for me) about this question. I'm working on a VAR model (7 time series) where i've checked for Granger causality (yes) and stationarity (yes). Now, according to the AIC ...
1
vote
0answers
46 views

ARMA zero coefficients (not statistically significant)

I seem to have forgotten what to do when the coefficients of the time series model are not statistically significant like $\phi_2$, $\phi_3$, $\phi_4$ below. A university course taught us to set each ...
1
vote
1answer
50 views

automatic model selection of linear models

https://i.stack.imgur.com/uYayd.gif https://i.stack.imgur.com/bAEMc.gif tail -c +43 uYayd.gif > TROW.tsv tail -c +43 bAEMc.gif > AABB.tsv Using the two ...
1
vote
3answers
77 views

How can we assume the models are exhaustive in Bayesian Model Averaging?

Bayesian model averaging is justified using the law of total probability which requires the the set of models that we average over to be exhaustive. Shouldn’t we prove that the set of models are ...
0
votes
0answers
52 views

Interpretation of Elastic net having too low or high value of alpha

Often I found the situation that the elastic model what I fitted has optimal alpha value at 0 or 1. Or not only that situation, but also there some alphas go near to 0 or 1.(ex. 0.1 or 0.9) My ...
1
vote
4answers
121 views

Thousands of features and only 70 samples

I am working on a regression problem where I have around 5-10 thousand features and have only 65 samples. I am training my algorithm on 55 samples and testing on 10 samples. I am using both Pearson ...
1
vote
1answer
107 views

Classical difference-in-differences: Coding the time (post) variable when treatment starts at different times

I have panel data with 40 treated cases and 40 control cases. I thought about the application of the 'classical' difference-in-differences (DiD) equation with the following linear regression model: $y ...

1
2 3 4 5
36