Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

Filter by
Sorted by
Tagged with
2
votes
0answers
14 views

R Squared (OOB) and R Square from correlation of prediction of test set is different?

I'm using simulated data and fitting Random Forest model for regression on a training dataset. What is confusing me is that after running Random Forest, I got R Squared (OOB) equal to 0.14. But when I ...
0
votes
0answers
11 views

Should I perform Tobit/Ordinal Regression instead of OLS?

I'm using regression to understand what demographic variables explain playtime and performance in an educational video game. Demographic variables are age is numerical while gender, race, income, ...
0
votes
0answers
13 views

Selecting best Causal Impact model from available set of different models [closed]

I am working on figuring out methodology to find best Causal Impact model from available set of different models generated by Causal Impact Library. Currently, I am exploring cross-validation approach ...
0
votes
0answers
30 views

Linear model selection (AIC/dredge function) with non parametric data (residuals are autocorrelated)

I used the dredge function with MuMIn & lme4 package to do a linear model selection with AIC principle. I have about 10 ...
0
votes
0answers
10 views

How to find the right parameters for the SARIMA model?

Here is the ACF and PACF graphs for differentiated series: From the ACF graph it looks like AR() model is more suitable, because the lags are decreasing exponentially and they show that the series ...
1
vote
1answer
32 views

Is BIC asymptotically efficient for minimizing prediction error if the true model is being considered?

If a set of models is being compared using BIC and AIC, given the fact that the true model (the one which generated the data) is in this set (and given the other assumptions that guarantee BIC ...
2
votes
1answer
30 views

Fixed effects in DAGs

Let's imagine I'm interested in studying the causal effect of beliefs in some ideas and behavior related to these ideas (say, if I believe sunscreen is good for my health, I use more sunscreen etc.). ...
0
votes
0answers
14 views

Interprete AIC as significance test and determine significance level for nested models

Let's say we have two nested models. The smaller one (corresponds to $H_0$) has $p_0$ parameters, so its AIC is given by $$AIC_0=-2\log L_0+2p_0$$ The larger model has $p_1:=p_0+d$ parameters of which ...
0
votes
0answers
19 views

Identifying $ARIMA(p,d,q)\times(P,D,Q)s$ process

I have a series of monthly data (top left) differenced (12) with lag = 1 (bottom left) with the following ACF (top right) and PACF (bottom right): And I'm trying to fit an $ARIMA(p,d,q)\times(P,D,Q)s$...
0
votes
0answers
17 views

How can compare suggestion models with different performances?

I have 4 class binary classification models. That models identify which class a particular students is suitable for. For example, we have user 1 and 4 classes ...
0
votes
0answers
8 views

Correct way to use cross-validation for hyperparameter selection when testing a model in multiple trials which use different train-test split

I'm going to evaluate a model with some benchmark datasets. I want to perform 100 trials of training and testing of the model for each dataset, and I want to randomly split the data again in training ...
0
votes
1answer
40 views

Selecting between models based on P-value [closed]

I have a general question regarding selection of regression models based on their P-values. I have two models, one with two independent variables and one with four. If the selection is to only be ...
1
vote
0answers
17 views

Split one model into two models for a classification problem

In the classification problem, I am working on right now, I have to classify transactions mainly with text data. The classes of the training set can easily be divided into class sets with a negative ...
0
votes
0answers
16 views

Separate Hyperparameter/feature selection and Model Selection cv vs nested cross validation (cv)

I realise that nested cross validation can be used to reduce bias when hyper-parameters tuning is combined with model selection. However, I wonder if it is possible to perform hyper-parameter tuning ...
0
votes
0answers
14 views

Stabilizing nested cross-validation

Quite recently I stumbled upon several posts here on nested cross-validation, which showed me how wrong my understanding was of such procedure. Now, trying to put all the pieces together, I still have ...
3
votes
1answer
33 views

Can I use AIC for path models and non-path models?

I am testing competing hypotheses where one hypothesis contains a mediation effects that can be modeled using a path model. The other hypothesis does not include a mediation effect and therefore can ...
0
votes
0answers
18 views

MultiClass Classification - Training OvO and OvA

I like to know how OvO (One vs One) and OvA (One vs All) models are trained in multiclass classification problem. To keep it simple, we have 4 classes, each of which has 1000 datapoints. What are the ...
0
votes
1answer
17 views

ANOVA for comparing same linear models

I have trained two linear regressor models with the same response $Y$ and the same features $x_1$ and $x_2$, they are basically the same model, however only the training data differs: in model1 data ...
1
vote
0answers
10 views

How should I check if there is interaction between variables in ANCOVA? and how I interpret the ggplot?

I was planning to test if GDP and fertility explain variation in average life expectancy and if the effects of these variables are dependent on the decade the data were collected and also which world ...
0
votes
0answers
23 views

How to determine the optimal lag order of asymmetric innovation for a GARCH?

In the python package arch_model, there is an option with which we can give the "lag order of the asymmetric innovation" when we estimate GARCH model. Are there any ways to find the optimal ...
16
votes
4answers
735 views

Why does the Akaike Information Criterion (AIC) sometimes favor an overfitted model?

As an exercise to develop practical experience working with model selection criteria, I computed fits of the highway mpg vs. engine displacement data from the tidyverse mpg example data set using ...
0
votes
1answer
48 views

Reducing a linear regression (OLS) model by dropping non-significant coefficients

Would it be proper for me to reduce a model by iterating though the coefficients and dropping the ones with high p-values and then refitting and doing this again until all coefficients are significant?...
0
votes
1answer
33 views

Parameter estimation of ARIMA model with exogenous variables (ARIMAX)

I am trying to compare an ARIMA model based on the price of a cryptocurrency without exogenous variables to one which adds in the number of tweets about the crypto in the same period as an exogenous ...
0
votes
0answers
19 views

Criteria for saving best model during training neural network?

I am doing 4-class semantic segmentation with U-net using generalised dice loss as loss function. General approach to save best model during training is to monitor validation loss at each epoch and ...
7
votes
2answers
476 views

Why is AIC or BIC commonly used in model selections for time series forecasting?

On scikit-learn documentation, I found the following comments about AIC: Information-criterion based model selection is very fast, but it relies on a proper estimation of degrees of freedom, are ...
0
votes
0answers
20 views

Is there something better than hyperparameter search

I am trying different arch-like models and different $p$, $q$ and $o$ values. If I fit, for example, GJRGARCH with $p=0, q=1, o=1$ I get better results than with any of: $p=1, q=1, o=1$; $p=0, q=2, o=...
0
votes
0answers
14 views

Model selection under different time window of data

I am adjusting historical monthly income amounts to an ARIMA model using auto.arima on R, with forecasting purposes. For that I have gotten different models based upon different time windows of the ...
0
votes
0answers
22 views

Orders of AR and MA models

I have a couple of ACFs and PACFs and cant seem to be sure about the order of AR(p) and MA(q).Can anybody kindly give me an insight into how to detect that?
1
vote
1answer
33 views

How to find the order for ARMA model?

I have a problem finding the order with the ACF and PACF plot, below is it. First I think they can be considered as tails off gradually because they are abnormal, then I set AR(1) from PACF and MA(1) ...
0
votes
1answer
30 views

Does the column ordering matter in the stepwise algorithms used by R?

Suppose I have a large data set with variables $x_1, x_2, \ldots, x_p$ to predict response $y$ where $p$ is very large (however $n >> p$). I would like to perform forward stepwise regression on ...
0
votes
0answers
12 views

Model selection and interaction terms in multivariate regression analysis of compositional data (Dirichlet regression)

I am currently exploring Dirichlet regression models to model fatty acid compositional data. I am using 2 categorical predictors and 1 continuous predictor (treatment group, sex, and total lipids). ...
1
vote
1answer
146 views

Convergence issues and model selection in glmmTMB

Convergence problems in mixed effect models seem to be a common struggle. It is my understanding that they emerge when the likelihood surface is too flat for the optimisation algorithms to find a ...
1
vote
2answers
42 views

Is the linear mixed-effects model the right choice for analysing my data?

I'm in desperate need of advice in terms of the choice of statistical test for my analysis. Briefly to explain what I am analyzing in an animal model: I want to see the effect of 2 categorical ...
0
votes
0answers
6 views

BIC for combined model selection on independent data sets (BIC not additive)

I have two statistically independent data sets, $x_1$ and $x_2$, both of size $n$, and I would like to select a model $m$ out of the same candidate set $\{1 \ldots M\}$ for each of them, i.e. I select ...
0
votes
0answers
12 views

When AIC chooses 2 lags, but 2nd lag is insignificant, do I drop it?

If the AIC criterion chose 2 lags, but the 2nd lag is not significant (see p-value), then am I supposed to drop the 2nd lag or leave it?
1
vote
0answers
39 views

Nested Cross validation with two settings on KNN

I am trying to perform model selection and evaluation using a 5-fold (internal) CV for the iris data. The things that I performed so far. Partitioned ...
1
vote
0answers
9 views

Which model to select of two similarly performing, models with similar architecture and number of parameters, but different depths

I am training U-Net models for two-class semantic segmentation (foreground/background). I have tested different depths of the U-Net along with different number of filters in the first conv-layer (the ...
0
votes
0answers
27 views

Selection of a “best” regression model using differet approaches

I need help with the following question. I am really lost, so any help/hint would be much appreciated! I am aware that for best fit model, we are looking for higher $R^2_{Adj}$, smaller $MS_{res}$, ...
0
votes
1answer
16 views

Should we use the same fitting method in model selection and prediction?

I am curious whether we have to use the same fitting method in model selection and prediction. For example, suppose that we are going to use the logistic regression in prediction. Then, we may select ...
0
votes
0answers
5 views

Subset Selection Methods for a Binary Response

The situation I face is as follows: 500 observations 100 predictors (features) A binary response What I want to do: Identifying a subset of the predictors that I believe to be related to the ...
1
vote
0answers
34 views

Can I always defend using ols, (for example when my dependent variable is ordinal), if I satisfy all CLM assumptions?

Previous reading Let me first say that I went through this post: (How to determine which distribution fits my data best?) and this post: Assumptions of linear models and what to do if the residuals ...
0
votes
1answer
34 views

What are the preliminary analysis before running a logistic regression?

I have a dichotomous variable which represents if a student is accepted or not in a University. In order to do this I have about 60 variables (information of the students: gender, age, etc; their ...
7
votes
1answer
92 views

GAM: Do shrinkage smooth splines also address for concurvity?

I have a gam model with automatic predictor selection based on cubic splines (bs = cr) and SELECT == T or shrinkage cubic splines (bs = cs) and SELECT == F. Now I'm wondering if predictors affected ...
0
votes
0answers
10 views

How do I determine p and q from an acf and pacf plot?

I know you determine p from and q from pacf and acf, but how do you find the optimal number? I still don't quite get how it's done after googling. What is the optimal p and q from the following charts?...
0
votes
0answers
19 views

identify tentative arima model from acf and pacf graphs

here are the ACF % PACF graphs for three different models, kindly please tell teh tentative ARIMA models using these graphs, I will be very much thankful to you
0
votes
0answers
12 views

Are hyperparameters chosen from cross-validation slightly biased towards greater regularization?

I intend to fit a single model to the entire dataset after selecting hyperparameters by k-fold cross-validation. So on each round of training, my model is fit to $\frac{k-1}{k}n$ of my dataset, and ...
1
vote
1answer
43 views

ARIMA (p and q values)

I am trying to do Arima forecasting, i differenced once so d=1, Im not sure what my p and q values need to be, please check screenshots of acf and pacf below:
0
votes
0answers
16 views

How do I know if my data is MA or an AR process?

how do I identify if the process is an ARMA, MA or AR process for example we have:
0
votes
0answers
15 views

Can I use AIC for model selection with same model on data subsets?

I have a class-imbalanced dataset so I divided my data into positive and negative classes (10% pos-90%neg). To model the data I planned to subset the negative data into 8 subsets and then create 8 ...
1
vote
1answer
42 views

Are there any key reasons as to why people would choose between apARCH, gjrGARCH and E-GARCH?

I’ve been doing a lot of R coding with GARCH for my dissertation, I'm coming to the end of my writeup now but have hit a bit of a wall. Obviously, gjrGARCH, apARCH and E-GARCH all allow for asymmetric ...

1
2 3 4 5
35