Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

Filter by
Sorted by
Tagged with
2 votes
1 answer
25 views

Does using grid search for hyperparemeters make test set redundant?

The purpose of train, validate and test data splits addresses the issue of data leakage when tuning for the model's hyperparameters. Does Grid Search then eliminates the need for test set? Because ...
user avatar
  • 21
0 votes
0 answers
20 views

R package for order selection of a vector ARIMA(p,d,q) for multivariate time series

I'm currently working on a project that requires dealing with multivariate time series. What I would like to fit to my dataset is a VARIMA(p,d,q) . I'd like to proceed using the Box-Jenkins procedure, ...
user avatar
0 votes
0 answers
21 views

Should we care about correlations between model performances while we search for the best model?

When we fit (train) a regression model, we usually pick the best performing model (for example the one which gives the smallest RMSE). By doing this we do not take into account the correlation between ...
user avatar
  • 473
0 votes
0 answers
22 views

ACF and PACF Plot

I am a first year stat student. We are tasked to create a SARIMA model from trial and error using ACF and PACF plot. Now here is my generated plot: Now I am trying to understand the plot but I don't ...
user avatar
  • 1
2 votes
1 answer
87 views

Why can't we use AIC and p-value variable selection within the same model building exercise?

In our assignment we were asked to model the compressive strength of concrete (response variable: Strength) with predictor variables Cement (kg/m^3), Water (kg/m^3), and Coarse.Aggregate (kg/m^3). We ...
user avatar
0 votes
0 answers
6 views

Fixed effect model selection when some candidate models have singular fits

I have an experiment on the performance of a new animal management technology. The response is a proportion and I have three continuous input variables. There is one grouping factor of interest (...
user avatar
  • 1
0 votes
1 answer
27 views

How to select predictor variables for linear mixed model?

I have a linear mixed model with ~30 clinical/treatment variables and repeated outcome variables for patients. E.g. The outcome variable is Breast symptom scores, which were collected at different ...
user avatar
0 votes
0 answers
25 views

Manual variable selection in GLMM

I'm modeling my data using GLMM with 1 random factor and 10 variables that are of interest. Instead of using automatic selection, I started with the full model (including all variables except for the ...
user avatar
1 vote
2 answers
59 views

Unintuitive results in model comparison

I am running an experiment for some time now and currently I am in the process of analyzing my data. At first I was unsure about which model fits my needs, but after receiving some much appreciated ...
user avatar
  • 23
0 votes
1 answer
51 views

Forcing covariates to always be part of a Lasso model

I want to use a Lasso to predict outcomes for different policy scenarios. At the optimal degree of regularization obtained by cross-validation, one important variable in whose impact I'm interested in ...
user avatar
  • 1
0 votes
0 answers
9 views

Bandwidth selection when comparing different local constant regression models

Let's say that I want to compare two alternative specifications of a Local Constant least squares model. If I had a single model, I would select my optimal bandwidth by cross validation. However, ...
user avatar
  • 1
0 votes
0 answers
8 views

Options for two-stage GAM univariate model selection when AICc values <2Δ

I am constructing a two-stage GAM (stage 1 presence/absence with binomial and logit link, stage 2 abundance with poisson and log link) to model capture rate across my study site. As this is a pilot ...
user avatar
  • 11
1 vote
1 answer
34 views

what does model complexity means in linear regression?

As far as I know, $y=\beta x$ is a not a complex model since we have a polynomial of the first order for all variables $x_i$. I am studying the linear the bias variance trade-off, and the lecture ...
user avatar
1 vote
1 answer
26 views

Threshold optimization with cross validation

I have an imbalanced dataset; 95% negative class and 5% positive class. I split my data into train (80%) and test (20%) sets. I am using 5-fold cross-validation on the train set to determine the ...
user avatar
2 votes
2 answers
351 views

AIC, BIC and log likelihood which more important?

I am currently searching for the best ARMA(p,q) model for my conditional mean. When comparing the AIC, BIC and LL, I see that some model perform better in AIC, some in BIC and some in LL. The AC and ...
user avatar
  • 41
1 vote
0 answers
25 views

Explaining AIC Change When Removing/Adding Variables

I might be getting turned around in my thinking of AIC. My understanding, an example: Let's say I have three models: 1.) AIC = 100, it has 10 variables 2.) AIC = 100, it has 9 variables 3.) AIC = 96, ...
user avatar
  • 11
1 vote
0 answers
24 views

Is it okay to use PCA then with LSTM

I am currently trying to see if there is any validity to this procedure: Using PCA on data in tabular data Then transforming the new "PCA data" into a multivariate time series Using an LSTM ...
user avatar
0 votes
0 answers
19 views

Best approach for AIC model selection?

I am doing a study where I am trying to model how different factors affect polar bear movement. I would like to conduct model selection using AIC. So far, I believe I have two options:   1)    Put ...
user avatar
  • 31
0 votes
0 answers
15 views

How can I do a comparative analysis of fbprophet time-series model results in pyspark? [closed]

I am using fbprophet for time-series forecasting on an unique ID of a big data with thousands of unique ...
user avatar
  • 101
1 vote
0 answers
39 views

What AR and MA values should I choose from my ACF, PACF plots

I have plotted ACF and PACF plots for my data, as I am trying to justify my hyperparameters chosen for my ARIMA model. The data is energy prices every day, with data taken every 30 minutes (48 points ...
user avatar
4 votes
3 answers
276 views

Practical justification for not basing model selection on performance on test data?

I've always been told not to use a model's performance on the test data to select a final model. I've read the responses to this question and others posted around the internet, but still have ...
user avatar
0 votes
0 answers
18 views

Model selection and validation of latent class models with sampling weights

We are conducting a LCA with sampling weights, using Stata. Since we apply these sampling weights, the dataset becomes very large and the goodness of fit tests do not validate the model and do not ...
user avatar
  • 1
1 vote
1 answer
14 views

Clarification on Danger of Inference on Predictors for Time Series Regression

I am reading the excellent book "Forecasting: Principles and Practice" and in chapter 7, section 5 there is the small section. Beware of inference after selecting predictors We do not ...
user avatar
0 votes
0 answers
27 views

Using SSE to find the optimal model does not yield the correct solution

I am generating an AR(3) model with coefficients $(\phi_1, \phi_2, \phi_3) = (0.35, 0.5, 0.08)$. Then I use the statsmodels module from Python and its Yule-Walker implementation to get back the ...
user avatar
1 vote
1 answer
55 views

Lag selection and model instability for ARIMA-GARCH in rolling windows for forecasting

I'm to produce rolling forecasts with an ARIMA-GARCH model using a moving window size of 1000. Given that structural changes in the series might take place at some point in the forecast horizon, is ...
user avatar
  • 23
1 vote
0 answers
146 views

Obtaining approximate posterior probabilities with Bayesian cross-validation

(Apologies to anyone that may have been following this question: I have decided to rewrite it to make it more succinct. As a result, comments below now appear out of context.) Given a set of models $\{...
user avatar
  • 1,135
0 votes
0 answers
26 views

GARCH model and variance equation

I have wheat prices in log 1st difference. I tested it for ARCH(p) effects, and ARCH effects does exists. So i built an GARCH(p,q) model. My issue is that I don't know which GARCH model I should use. ...
user avatar
0 votes
0 answers
16 views

regression model selection method

Are there any academic papers that used MSE*p to select model from a set of candidate models (for example, candidate models are from backward selection)? MSE is mean squared error, and p is the number ...
user avatar
  • 1
0 votes
0 answers
46 views

Paths to optimal K for GAM model selection

Let's say I have 10 different model combinations to compare via AIC for one year. There are 3 years of data, roughly 200-400 observations each year. For covariates, 2-3 of 5 appear to require tweaking ...
user avatar
0 votes
0 answers
45 views

Is correct to apply Wilcoxon Signed-Rank Test on multiple datasets?

I am learning about statistical tests and trying to apply this concept in machine learning by comparing different classifiers in multiple datasets. As for my understanding, to compare two models in ...
user avatar
0 votes
1 answer
39 views

How to optimise ARIMA model results - Have p q d been selected properly

I'm having trouble interpreting my ARIMA results and would like to know what I can do to improve the model. I am working with a crime data set with 7740 entries collected over 21 years which I've ...
user avatar
  • 1
0 votes
0 answers
20 views

Why Choose IGARCH over Standard GARCH?

I understand that IGARCH is a nested version of the standard GARCH model where alpha+beta=1, which implies a unit root. Although, I am struggling to see why having a GARCH process with a unit root is ...
user avatar
1 vote
0 answers
39 views

Distribution of Bayesian Information Criteria - using BIC when there are multiple datasets

I have two competing nested models : $M1$ and $M2$. $M1$ has way less parameters. I know that $M2$ is definitely a true model for data (i.e., can explain data fully) but I claim that it is not the ...
user avatar
  • 31
0 votes
0 answers
17 views

model to explore correlation of pollution long-term exposure with genetic mutation rates

I have pollution data of several US cities, as example NY, Boston, and Chicago. I need to ...
user avatar
  • 207
0 votes
0 answers
8 views

Choosing between a set of potential instruments

I have a question regarding instrumental variable analysis/2SLS. I would like to estimate the following IV regression: $$Y_{ht} = 𝛼_{h} +𝛿_{𝑡} + 𝜷_𝟏(MM_{ht})+ 𝜷_𝟐(Shock_{ht−1})+𝝉(MM_{ht}∗...
user avatar
0 votes
0 answers
25 views

Can we compare whether different groupings of data improve model accuracy?

I have data from 100 different lab incubations of manure samples. For each sample, a 3-day incubation was done, measuring values (y-axis) against time (x-axis). I want to perform a linear regression ...
user avatar
0 votes
0 answers
15 views

Seeking A Scale-Independent Alternative To Q2 For Model Selection When The Response Varies Over Multiple Orders of Magnitude

I am using constrained polynomial regression to predict y = f(x). I have prior knowledge about the relationship that allows me to add constraints to the optimization problem for the first and second ...
user avatar
0 votes
0 answers
21 views

Should both "Student ID" and "Quiz" be random effects in a mixed effects model?

I'd like some help constructing my model, please. Research question: Does the rate of homework completion affect student quiz scores. My data is at the student-chapter level. This means I have chapter-...
user avatar
  • 121
0 votes
0 answers
23 views

How to approach insignificant coefficients in time series?

Let's say we have an ARIMA model. The series is stationary, autocorrelations significant etc. However, the p-values of the coefficients suggest that the coefficients are insignificant. Now, I've heard ...
user avatar
  • 121
1 vote
0 answers
52 views

How to determine whether to model GARCH effects over ARCH effects?

Based on my understanding, we could determine whether or not to include ARCH effects, by checking the residuals of the mean corrected model (EX: ARMA model). I know the difference between GARCH ...
user avatar
3 votes
1 answer
63 views

Controlling for phylogeny in linear models

I have run a linear model to test for a relationship between two continuous traits in a sample of 50 taxa. I'm using the packages phylolm and caper in R (just to check they tell me the same thing, ...
user avatar
  • 143
0 votes
0 answers
16 views

Final model training after cross validation for a regularized neural network

I have a regression problem with a relatively small number of available samples. In order to select the best model and tune the hyperparameters, I am using nested cross-validation (NCV). The common ...
user avatar
1 vote
0 answers
24 views

Single/ multiple imputation in post-selection/-regularization context

Context of problem: In some situations researchers face high-dimensional problems with $p > n$, where $p$ is the number of covariates to be considered in a regression model and $n$ is the sample ...
user avatar
  • 327
0 votes
0 answers
27 views

Comparing nested regression models on multiple data sets

Suppose we have two nested models $M_1(p_1)$ and $M_2(p_2)$ where $p_1$ and $p_2$ are the parameter sets of the models. Consider that $M_1$ is a restricted version of $M_2$ so $p_1$ set is a special ...
user avatar
  • 31
10 votes
4 answers
864 views

In what applications do we prefer Model Selection over Model Averaging?

I'm wondering in what applications or scenarios (or in trying to answer what kind of questions), the researcher would prefer using Model Selection (such as AIC or BIC) over Model Averaging (such as ...
user avatar
  • 1,682
0 votes
0 answers
23 views

What alternative techniques are available to perform variable selection with big data?

I have a 88x6500 dataset of people income, where 88 are time periods (quarters) and 6500 are people. My independent variable is X = income and my dependent variable is Y = average houses price in a ...
user avatar
  • 67
13 votes
4 answers
1k views

How simple should a Baseline model be?

I was making baseline model and was wondering how much time I should spend on it. I've found a lot of article about the purpose of a baseline model, for instance; A baseline takes only 10% of the ...
user avatar
1 vote
0 answers
26 views

Could I pick the best model based on the bias-variance tradeoff?

Usually would I pick the best performing model according to accuracy, or another evaluation method, on the validation dataset. But is this viable to chose the best model according to bias-variance ...
user avatar
0 votes
0 answers
10 views

What is the number of free parameters in an n-component GMM?

I am trying to calculate BIC = -2logL + log(N)d where d is the number of free parameters or degrees of freedom. If I am fitting guassian mixture model to the data, ...
user avatar
  • 1
1 vote
1 answer
55 views

What is the purpose of the model AIC if you can iteratively check the forecast RMSE at each lag?

If the purpose of the time series modelling is to build one that gives the most accurate forecast, may I ask if it necessary to check the model AIC to determine the optimal lag when you can ...
user avatar

1
2 3 4 5
37