Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

1
vote
1answer
12 views

Clarifying lag number selection in AR,VAR, VECM etc. models

When it comes to optimum lag length selection, we are supposed to comply with certain information criteria such as Akaike, Schwarz etc. As far as I know, either of them suggest the proper lag number ...
3
votes
1answer
30 views

Ockham's Razor in Bayesian Modelling

This question might be a little philosophical / generate discussion. I hope, there may still be some useful answers. I am currently thinking about how Ockham's Razor relates to Bayesian statistical ...
2
votes
1answer
19 views

Binomial distribution as likelihood in Bayesian modeling. When (not) to use it?

I am currently trying to figure out some strangeness about using the Binomial distribution in Bayesian modeling to define the likelihood. To make an example assume I have two conditions, and in each ...
1
vote
0answers
22 views

Regression with a high number of 0's for target variable. How do I approach this?

I have a dataset where the probability of an event happening is very low (15%-20%). When the event happens, there's a dollar amount attached to it. The distribution is very right skewed, ranging from -...
3
votes
2answers
115 views

ACF and PACF of residuals to determine ARIMA model

I'm having trouble interpreting an ACF/PACF plot of the residuals of a regression to determine what the corresponding ARIMA model would be for the error term. This is the plot of the ACF/PACF of the ...
2
votes
0answers
33 views

When to disregard AIC as a criterion in model selection

I have the following problem: I'm working on a dataset and it looks completely quadratic. A quadratic regression fits the data really good. However, when using piecewise linear functions I get a lower ...
0
votes
0answers
15 views

strictly nonnested models vs partially nonnnested/overlapping - Vuong's test

I'm trying to use Vuong's test to compare LR models - having trouble with the definition of 'strictly nonnested' vs 'partially nonnnested'/overlapping. There are a few (lay language) definitions of ‘...
1
vote
2answers
133 views

Cross-validation and building a final model when using hyperparameter optimization

I am trying to build a Gaussian process (GP) regression for a problem in which each experiment is computationally expensive, using cross-validation. Here is how I do it: Build the GP regressor on the ...
0
votes
0answers
7 views

Select model based on multiple performance metrics

I have a few performance metrics - MAE, RMSE, and MAPE. I choose my model hyperparameters on the validation set using MAE so far. However, I compare models among themselves on the test set using all ...
4
votes
1answer
44 views

Why is the deviance defined with a factor 2 (or likelihood ratio squared)?

Deviance is defined as I see the motivation in why we would define the deviance as a difference of logLikelihoods or just the log(Likelihood Ratio), but why the factor 2? Why square the ratio? Does ...
1
vote
0answers
33 views

Approximate bayesian computation: model selection on nested models

For model selection within an ABC framework when the models are nested, say model 1 is equal to model 2 on some subset of the parameter space, is it better to try and do parameter inference or use a ...
0
votes
0answers
11 views

GBDT- randomized repetition feature selection

Consider the following approach for feature selection in the specific case of gradient boosting decision trees: Randomly pick X% of features Run algorithm Record importance of each feature Repeat ...
2
votes
1answer
64 views

Selecting between OLS regression and ARIMA for time series, why AIC or BIC for ARIMA is much larger in my data?

My data set is quarterly time seires data (around 140 data points). Method 1: simple OLS regression with 5-6 exogenous variables, which are drivers of the dependent variable. None of the explanatory ...
1
vote
1answer
27 views

How many candidate models to include in AIC model selection?

Is there a rule of thumb, perhaps related to sample size, for how many models to include in AIC model selection? Too many may seem like fishing while too few would be insufficient. I'm familiar with ...
0
votes
0answers
43 views

Selection of dynamic regression model differencing based on cross-validation

I have got confused a lot. Suppose I have a time series, which is non-stationary with high probability (I cannot plot it, since there are 1000s of them). I need to fit regression with error being ...
1
vote
0answers
30 views

Model selection using p-values - tree inference

Suppose I have some i.i.d. normal observations from $\mathbb{R}^f$ with parameters $(\mu, \Sigma)$ and $\Sigma$ is known to be the identity matrix. I have the following hypotheses: $H_0^i$: $\mu_i = ...
0
votes
0answers
24 views

VAR/VEC model selection

I want to model a relationship between a good's price and a few variables using time-series data. I run VEC/VAR models and get a series of equations. My question is how to use these results (using ...
1
vote
0answers
32 views

What is wrong with this model selection procedure?

I have a set of ~400 observations and ~20 covariates. Some covariates are logged, sqrt'd or truncated versions of others, so lots of dependence in my model matrix. My response is a proportion. I ...
0
votes
0answers
24 views

Handling collinearity in GAM

I am interesting if a variable x1 is of importance for the outcome y. To investigate this I am trying to fit a model A1 <- gam(y~ s(x1) + s(x2) + x3 + x4 + x5 + x6)) where x1 and x2 are ...
0
votes
0answers
14 views

How to do a right model selection for modeling decision trees in R

I struggle with modeling a decision tree. I am trying to model both types (classification and regression), but let's stay for the first with regression tree. I do have a large data set of 200000 ...
0
votes
1answer
21 views

AIC calculated in lm(y~1) and stepwise selection in R

http://www.stat.wisc.edu/courses/st333-larget/aic.pdf The AIC calculated with the model lm(SAT~1) was 560.4736, but the AIC calculated with stepwise selection starting with lm(SAT~1) was 419.42. May ...
0
votes
0answers
14 views

Selection of regression model for prediction and interpreting quadratic regression results

I am regressing between the body mass and eye diameter in different bat species. The relationship is non-linear (picture attached) as the eye-size cannot increase linearly with respect to body size ...
0
votes
0answers
26 views

Shrinkage methods - are they any good for statistical inference or should they be used for prediction goals only?

I am working on my master thesis with a goal to find regressors which influence companies' decisions on how to pay for a target in acquisitions (cash, stock or a mix of both). I have 13 regressors to ...
0
votes
0answers
18 views

VAR model residual autocorrelation and variable selection

I have a question on VECM model. I have a set of variables I had planned to include in my VECM model where one particular variable may be trend stationary (@ 10% s.l. by ADF test) while the rest are ...
0
votes
1answer
45 views

Random Forest % Var explained OOB output differs from test dataset results

I am learning how to use Random Forest in R for regression based on the Boston dataset. I am unsure on which values I should concentrate to evaluate the obtained model, the OOB % Var explained and MSE ...
0
votes
0answers
20 views

LR test for VAR model selection: p value goes increases and then decreases

I have a question on VAR model using LR test to select the lag lengths. My result is shown below and you can see that LR test rejects lag 4, so seemingly I should use lag 3. But then lag 5 has p-value ...
1
vote
0answers
73 views

Multivariate ARIMA modelling in R

I am currently using the Marima package for R invented by Henrik Spliid in order to forecast multivariate time series with ARIMA. Overview can be found here: https://cran.r-project.org/web/packages/...
0
votes
0answers
26 views

Determining which variables to use in regression model

So I'm trying to fit some binary outcome data to a logistic regression model. Besides the binary outcome I have several different metrics (numeric, integers, as well as factors) associated with each ...
5
votes
0answers
38 views

Why does “mixtools” return the model with highest AIC as the “winner” if lower AIC is better?

Mixtools package is used to fit mixtures of normal/regressions. The package documentation is given here The regmixmodel.sel fits the mixture model for varying ...
3
votes
1answer
71 views

ARIMA - What is the proper ARIMA model for these data?

I am doing my project on forecasting and due to I have limited knowledge in ARIMA, I would like to ask what is the appropriate ARIMA model for these two data. Both data are monthly. Figure 1 The ...
2
votes
1answer
45 views

Is it possible (or even advisable) to only include random slopes for some contrasts of levels of a factor but not others?

Suppose the following model: DV is reaction time. The predictor is a categorical factor with three levels, manipulated within participants. Each participant gets fifty trials at each level of this ...
0
votes
0answers
25 views

What ARIMA to use

I have a data set which generally decreases over 24 period units. It then returns to its relatively highest state at the beginning of the period. So for instance the data may look like this: Period 1 ...
1
vote
0answers
37 views

How to choose different number of Neurons per Hidden Layer?

The post about how to choose the amount of hidden layers and neurons was extremely helpful. The rules of thumb given gave me often a good point to start. However, I'm now thinking about varying the ...
0
votes
0answers
10 views

Model selection for mixture of regressions - what order?

Suppose we are fitting a mixture of regressions. That is, $y_i$ is assumed to come from a mixture $$f(y) = \sum_k \pi_k f_N(y|\mu_k, \sigma_k )$$ Where $$\mu_k = \beta_k^T x$$ and $f_N$ are ...
2
votes
0answers
44 views

Zero-inflated highly skewed predictor variables

I've thoroughly searched this website and multiple others and can't seem to find an answer to my question. This is also my first post so I hope I've followed all the rules. I apologise for the length, ...
0
votes
0answers
5 views

Low t Values Multicollinearity

I'm studying a model which has large datasets of empirical data and analysis available. In the particular dataset that I consideted (with a few a priori expectations) I tried out a simple linear ...
0
votes
0answers
31 views

Aikaike Information Criterion: derivation in original paper

I have been reading AIC paper 'Information theory and an extension of the maximum likelihood principle' by Akaike (1974). I have been able to understand up to the third section of the paper, but I am ...
0
votes
0answers
48 views

Convince me that AIC can't be used to compare models with different sample sizes

Conventional statistical wisdom says we cannot compare AIC (or other information criteria predictive sample statistics) when the sample sizes for the compared models are different (See here for ...
1
vote
0answers
19 views

What is the point of using PRESS instead of RMSECV?

What is the point of using predicted residual sum of squares (PRESS) instead of root-mean-squared-error-of-cross-validation(RMSECV)? In many books, especially in the area of chemometrics, the authors ...
0
votes
0answers
33 views

How to pick models for long tailed ACF/PACF?

This is a real data ACF/PACF plot: You can see there are long tails in the plot. So any hints to select ARMA(p,q) model, or even ARCH model? Tried a few ARMA(3,3,0) or ARMA(3,1,0), apparently didn't ...
1
vote
0answers
28 views

Linear regression AIC and Randomisation Test

The problem is that I used AIC as the criterion for model selection and that gives me a model with 3 parameters (the model has the lowest AIC). $$y = \beta_0 + \beta_1X_1+\beta_2X_2+\beta_3X_3$$ ...
1
vote
1answer
44 views

How to deal with Errors which are not following Normal distribution?

I am working on Model Selection problem, where there are two models which are predicting revenues for companies using 2 different formulas. So currently I have Actual revenue values and predicted ...
0
votes
0answers
11 views

Bayesian information criterion (BIC) on KDE?

Consider two datasets, $X$ and $Y$. Both have 2 dimensions with $a$ and $b$ samples respectively. I would like to test whether one kernel density estimate (KDE) on the concatenated data ($XY$, shape $...
1
vote
1answer
46 views

More features, less F-Score

Is there any rule about relationship between number of features and performance of the model? Recently, I did an experiment on 3 sets of features (all extracted from a same dataset). The strange point ...
4
votes
1answer
45 views

In linear regression, data is highly skewed, transformation doesn't work..!

I have dataset with 9524 observations / 97 variables. Most of variables are numerical, and some of factor variables (Yes/no or several levels) I want to perform multiple linear regression with ...
1
vote
0answers
6 views

Why testing after selection?

If in model selection a model is chosen based on an estimation of the risk function, why do we need to retrain the selected model and do an assessment on the test set? Can't we use the estimation done ...
0
votes
0answers
31 views

Different cost of False Positives and False Negatives

In a classification task, I'm using GridSearchCV for model selection and evaluation. Given condition, that each False Positive (FP) event costs ...
0
votes
0answers
36 views

Using test MSE to compare models

I was reading through the definition of MSE and the formula I have found in all articles is the one shown in https://en.wikipedia.org/wiki/Mean_squared_error , which is the expected squared difference ...
1
vote
0answers
21 views

Model Selection and inference for mixture of logisitc regressions (or GLM) with heterogenous covariates by component

I am facing a problem which should be quite common IMO but for which I don't find relevant contribution. So the situation is this. Let's say that a binary response $Y$ is generated by a mixture of $K$ ...
0
votes
0answers
14 views

Cointegration vs regression of differences

I have a small number I(1) time series (under 10) that are cointegrated. I would like to create a forecasting model and my choices are either cointegration or regression of differences. I understand ...