Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

0
votes
0answers
12 views

Deduction for join two non linear equations [on hold]

My mathematics skills are not strong. I'd like to make the deduction to join two non-linear equations. I have one equation $Y_v = \beta_0 + x_1^{\beta_1} + x_2^{\beta_2}$ that explains $Y_v$ as a ...
1
vote
0answers
16 views

Selecting lag length for VAR Model. Differences or Levels?

I'm currently testing for optimal VAR lag length using the information criteria. I found that my variables are non-stationary (i.e. they have to be first differenced). When I identify the number of ...
0
votes
0answers
9 views

Tuning distance threshold in online clustering

In online clustering there are approaches where a threshold $r$ on the distance to the nearest cluster is used to determine whether a new data point should be associated to an existing cluster or ...
0
votes
0answers
15 views

Ranking Localities using Regression [closed]

I have a data-set which consists of multiple localities along with various features and I would like to rank each locality based on input features. Sample Data ...
1
vote
0answers
51 views

Determination or AR and MA parameters

I have in my possession price and time of different trade from an auction. The price series isn't stationnary so I work with the log return series. I'd like to forecast the evolution of the log return ...
0
votes
0answers
25 views

Confused with regsubsets() in R

I am a little confused with regsubsets in R, and the different methods- AIC, Cp and Adjusted $R^2$ in general. Suppose our model has $p$ predictors. From the output ...
0
votes
0answers
21 views

How to estimate model predicted means per group from a GEE model fitted in R?

This question is related to a similar post: How can I estimate model predicted means (a.k.a. marginal means, lsmeans, or EM means) from a GEE model fitted in R? The difference is that I do not only ...
0
votes
1answer
32 views

What model to use [closed]

I have data on whether an audition was successful or not and some data that could help explain the success/failure up to some extent (I have about 10 categorical variables and 2 continuous). So the ...
0
votes
0answers
11 views

What is a “subclass of linear models”?

I found the definition of graphical model structure in a paper, and it used the term "a subclass of linear models has graphical model structure if ...". What exactly is a class and a subclass of ...
1
vote
2answers
110 views

Time series Forecasting in Python - comparing different models

I have a large number of different timeseries and I need to create a forecast for each one of them. Are there packages that enable: Auto tuning of models? Cross validation of different tuned models -...
2
votes
1answer
35 views

K-NN optimal value of 'K'

How can I find the optimal value of 'K' in K-NN using the cross_val_score function, with scoring metric as auc_score? Do I need ...
12
votes
4answers
239 views

What is the point of univariate regression before multivariate regression?

I am currently working on a problem in which we have a small dataset and are interested in the causality effect of a treatment on the outcome. My advisor has instructed me to perform a univariate ...
2
votes
2answers
57 views

Which ARIMA model do I have to select?

I have monthly data since 1990 to January 2019 and I need to make a forecast of the next 6 months so I use a sample of the past ten years(I used a sample of the last ten years due to my job ...
3
votes
2answers
108 views

What's the real purpose of cross validation?

As for cross evaluation (CV), I have two questions to ask: 1) CV has nothing to do with parameter selection, but only model evaluation? Specifically, which model? 2) In k-fold CV, what's the final ...
2
votes
0answers
48 views

Model selection with different fixed effects and different corARMA structures

I analyzed the effect of temperature (4 different areas) on laying date: LDT ~ Aa3+Bb+Cc+Dd. Because of autocorrelation in residuals I used ...
1
vote
1answer
37 views

How do you select the best from a number of linear models?

I'm learning linear regression with the Carseats data set. I went through the data, cleaned it, encoded the dummy variables and checked for collinearity. The dataset has 400 observations on Carseat ...
1
vote
1answer
31 views

ABC, compute Bayes factor from posteriors

I am pretty new to ABC stuff so I may be saying dumb things. My question is: I ran an ABC with two models $M_1$ and $M_2$ and now I have an approximation of the posterior distribution for both model. ...
1
vote
0answers
10 views

Average time series forecast errors from cross-validation with rolling origin

I'm calculating the MAPE and RMSE over a rolling origin cross-validation with fixed forecast interval for several models. For example, for a daily series with 3 years, I'm training my model with 2 ...
0
votes
0answers
19 views

Bias correction when using loo cross-validation to replace unreliable PSIS-LOO estimates

The PSIS-LOO information criterion (see this paper by Vehtari, Gelman, and Gabry) assigns a Pareto shape parameter $\hat k$ to each observation in the data, and these $\hat k$ values can be used to ...
1
vote
1answer
25 views

ANOVA selects a model with autocorrelated residuals

I want to know which temperature dataset (Aa1, Bb, Cc, Dd) is/are the best predictor for laying date (medini). First, I used simple linear regression: median...
0
votes
1answer
23 views

Can overfit happen in spite of validation and what to do with it?

Let's consider a standard situation where we need to find a predictive model. We train all the available model using a training data set. We validate all the trained model using a validation data ...
0
votes
1answer
13 views

How does train-validation-test procedure deals with the sampling error of the accuracy measure?

Let's consider a standard model selection procedure: We have N different untrained models (for example linear regression, neural network, decision tree and so on). We use a data set A to train each ...
0
votes
0answers
23 views

Are there two motivations for Bayesian information criteria?

Are there two motivations for all these Bayesian information criteria? I am only aware of the motivation of "expected out-of-sample prediction score." Let the in-sample data be $y$ and the parameter ...
1
vote
0answers
13 views

What is a good test for whether a sample is drawn from a particular parametric family against a generalized alternative

Suppose I have some large number n of draws from a strictly positive distribution that I believe to be a member of a particular parametric distributional family. I use the draws to estimate the ...
0
votes
0answers
25 views

Generalising AIC results over multiple samples

This is slightly related to my previous question (AIC Calculation using log likelihood) Though, I think now I am actually clear as to what I am asking. I am modelling activity of cells, I have data ...
0
votes
0answers
14 views

Can we mix the algorithm?

The question comes from when I was wondering about the cross-validation, finding out the best algorithm. Then I got the question like if this model did a better job than the other, can the ...
2
votes
0answers
43 views

Why not use Ridge after Lasso vs relaxed Lasso

Has anyone ever applied a ridge regression on a model subset selected from a cross validated lasso? In other words, take a data set with p features and run lasso, grid searched to find optimal ...
1
vote
0answers
10 views

Should lower and upper bounds for a distribution count as parameters in AIC model selection

Suppose we want a random variable $X$ to be constrained and thereby to lie within specified bounds other than the natural bounds of the underlying distribution. This should be understood such that if ...
0
votes
1answer
45 views

cross-validation analysis not diagnostic

I'm using k-fold cross-validation analysis for model selection, however, it does not appear to favor any particular model. There are several variants of the models and two of them are nested within (...
2
votes
1answer
35 views

Determine paramaters for SARIMA model

I have the following timeseries with a frequency of 12 (months). Since there is both a trend and seasonality, I differenced the timeseries. To determine the parameters p, q, P and Q for the SARIMA(p, ...
1
vote
1answer
23 views

Distinguishing between overfitting and wrong model selection

I built a dozen of different models using caret package for classifying customer purchase habits into 5 categories (catA, catB, catC, catD, none) based on 4 numeric ...
0
votes
0answers
13 views

Setting Average posterior probability value in Stata traj plugin

In complement to Aikake Information Criterion, I want to use posterior probability to select the best model for group-based trajectory modeling. In this paper: https://drc.bmj.com/content/bmjdrc/4/1/...
2
votes
2answers
52 views

Comparing two regression model (Beta regression and linear regression)

I was informed that beta regression was more preferred to be applied to proportion data instead of linear regression. I know I can use various ways to compare the goodness of fit of two models, such ...
0
votes
0answers
36 views

ARIMA lag order selection by auto.arima

I use the funtion auto.arima in my dataset auto.arima(Clean_Ts_Dados,trace = T, stepwise = F, approximation = F) and got ...
1
vote
0answers
50 views

Estimating lag order in Granger causality test

I have a weekly revenue from selling products, named Chicken and Egg. I am trying to understand whether purchasing Chicken Granger-causes customers to buy Egg or vice versa. I don't have a Ph.D. in ...
1
vote
0answers
30 views

What is the most common KIC? How does it work?

Information Criterions are methods of assessing model fit penalized for the number of estimated parameters. Another question on the site asked for a comparison between the KIC and two other common ...
0
votes
2answers
34 views

Calculating the relative likelihood with AIC values

I'm using AIC for model selection, and would like to use a relative likelihood measure to quantify how many times a model with minimum AIC (AICmin) fits better than the alternative (with AICi). For ...
1
vote
1answer
36 views

Clarifying lag number selection in AR,VAR, VECM etc. models

When it comes to optimum lag length selection, we are supposed to comply with certain information criteria such as Akaike, Schwarz etc. As far as I know, either of them suggest the proper lag number ...
4
votes
1answer
43 views

Ockham's Razor in Bayesian Modelling

This question might be a little philosophical / generate discussion. I hope, there may still be some useful answers. I am currently thinking about how Ockham's Razor relates to Bayesian statistical ...
2
votes
1answer
85 views

Binomial distribution as likelihood in Bayesian modeling. When (not) to use it?

I am currently trying to figure out some strangeness about using the Binomial distribution in Bayesian modeling to define the likelihood. To make an example assume I have two conditions, and in each ...
1
vote
0answers
33 views

Regression with a high number of 0's for target variable. How do I approach this?

I have a dataset where the probability of an event happening is very low (15%-20%). When the event happens, there's a dollar amount attached to it. The distribution is very right skewed, ranging from -...
4
votes
2answers
275 views

ACF and PACF of residuals to determine ARIMA model

I'm having trouble interpreting an ACF/PACF plot of the residuals of a regression to determine what the corresponding ARIMA model would be for the error term. This is the plot of the ACF/PACF of the ...
2
votes
0answers
40 views

When to disregard AIC as a criterion in model selection

I have the following problem: I'm working on a dataset and it looks completely quadratic. A quadratic regression fits the data really good. However, when using piecewise linear functions I get a lower ...
0
votes
0answers
18 views

strictly nonnested models vs partially nonnnested/overlapping - Vuong's test

I'm trying to use Vuong's test to compare LR models - having trouble with the definition of 'strictly nonnested' vs 'partially nonnnested'/overlapping. There are a few (lay language) definitions of ‘...
1
vote
2answers
157 views

Cross-validation and building a final model when using hyperparameter optimization

I am trying to build a Gaussian process (GP) regression for a problem in which each experiment is computationally expensive, using cross-validation. Here is how I do it: Build the GP regressor on the ...
0
votes
0answers
10 views

Select model based on multiple performance metrics

I have a few performance metrics - MAE, RMSE, and MAPE. I choose my model hyperparameters on the validation set using MAE so far. However, I compare models among themselves on the test set using all ...
4
votes
1answer
70 views

Why is the deviance defined with a factor 2 (or likelihood ratio squared)?

Deviance is defined as I see the motivation in why we would define the deviance as a difference of logLikelihoods or just the log(Likelihood Ratio), but why the factor 2? Why square the ratio? Does ...
2
votes
0answers
37 views

Approximate bayesian computation: model selection on nested models

For model selection within an ABC framework when the models are nested, say model 1 is equal to model 2 on some subset of the parameter space, is it better to try and do parameter inference or use a ...
0
votes
0answers
20 views

GBDT- randomized repetition feature selection

Consider the following approach for feature selection in the specific case of gradient boosting decision trees: Randomly pick X% of features Run algorithm Record importance of each feature Repeat ...
2
votes
1answer
68 views

Selecting between OLS regression and ARIMA for time series, why AIC or BIC for ARIMA is much larger in my data?

My data set is quarterly time seires data (around 140 data points). Method 1: simple OLS regression with 5-6 exogenous variables, which are drivers of the dependent variable. None of the explanatory ...