Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

2
votes
0answers
23 views

$\text{Var}(\hat{y})$ versus $\text{Var}(y)$ for a general fitting algorithm

Suppose we wish to produce from a training set $(x_1, y_1), ..., (x_N, y_N)$ a fitted value $\hat{y}_i \approx y_i$. Let $\sigma_y = \sqrt{\text{Var(y)}}$ and $\sigma_{est} = \sqrt{\text{Var}(\hat{y})}...
0
votes
0answers
10 views

AIC criteria for a matrix decomposition problem

I am trying to decompose a matrix such that $$A \approx UV_1 \approx UV_2V_1 \approx UV_3V_2V_1V_2$$ where $A \in R^{n \times l}$, $U \in R^{n \times k_1}$, $V_1 \in R^{k_1 \times l}$, $V_2 \in R^{k_2 ...
2
votes
1answer
30 views

Cross validation and over-fitting

I've read many posts on this site that claim something along the lines of "I used cross-validation to prevent over-fitting". Which leads me to my question, does cross-validation actually prevent over-...
0
votes
0answers
9 views

Model selection with this model of a large number of components

I have a discrete time Markov Chain $\{X_n: n \in \mathbb{N}_0\}$ with unknown transition matrix $P \in \mathbb{R}^{M \times M}$ on the state space $\mathcal{S}_X = \{1,2, \dots, M\}$, with $M \geq 2$....
1
vote
1answer
19 views

Forward Model Selection Using p-value

I know that it is not advised to use the p-value as the criterion in practice, but I am not asking about that. I am wondering how this p-value would actually be calculated. In other words: Forward ...
0
votes
0answers
25 views

Is it useful to compute R Squared for regression trees? [duplicate]

I have a regression tree and want to validate the peformance. The first measure I have is the mse to find which model is the best. After that I want to check if the model peforms better then an ...
3
votes
1answer
49 views

AIC for increasing sample size

I am using AIC as a model selection criteria in one of my projects. However, since AIC isn't dependent on the number of points sampled, for large n the log likelihood term rapidly outscales the ...
1
vote
0answers
29 views

R-squared vs MSE, why the discrepancy?

I am carrying out a project where I am imputing missing data. I am trying to compare an imputed dataset with a baseline dataset by measuring MSE and R-squared. These metrics are measured by ...
0
votes
1answer
11 views

Correlation Analysis and Data Leakage

In machine learning, we perform feature engineering and selections in pipelines and crossvalidate to obtain results in order to avoid data leakage and avoid introducing prior knowledge into the ...
0
votes
1answer
32 views

Will reducing the number of explanatory variables guarantee an increase in training error?

It is known that increasing the complexity of a regression or classification model reduces the training error ( see e.g. Elements of Statistical Learning Chapter 2 ). My question is whether the ...
2
votes
0answers
23 views

Model selection for this model with one observation

I would like to perform model selection given a range of $k$ models $\mathcal{M}_1, \mathcal{M}_2, ..., \mathcal{M}_k$, each with some prior probability $f(\mathcal{M}_1), \dots, f(\mathcal{M}_k).$ ...
0
votes
0answers
8 views

Interpreting discrete choice experiments with sequential options

Objective: We have fruit fly stocks of different eye-pigment phenotypes, with different visual acuity levels. We want to know if, given a number of environments with different lighting conditions, ...
1
vote
1answer
43 views

Vector Autoregression - How do we choose the correct value of p?

I am following this article: https://otexts.com/fpp2/VAR.html#fn24 ...
2
votes
0answers
53 views

Is there a measure of “complexity” for linear/nonlinear model terms?

My apologies if this is grossly misunderstood or mis-worded, but I've been mildly bugged by a question to which I've not found a satisfactory answer. I can't say that I have seen a discussion about ...
1
vote
0answers
35 views

AIC for Causal Inference

I read a post explaining why the Akaike Criterion cannot be used for deciding if A cause B or B caused A. I'm curious about a more general case of using AIC for causal inference (with observational ...
0
votes
2answers
88 views

How to find all models that meet the pre-specified restrictions

Let's say I have a large number of predictors (e.g. 2000) and I'm facing the problem of choosing the linear regression model under following assumptions: There are few predictors that have to be ...
0
votes
0answers
14 views

eviews augmented dickey fuller lag selection

Can someone tell me how does eviews calculate teh optimal Schwarz lag selection? I did a quick search this https://en.wikipedia.org/wiki/Bayesian_information_criterion is this the same method that ...
1
vote
1answer
23 views

How can i choose the optimal lag in GARCH-MIDAS?

I have to choose individual GARCH-MIDAS models for some variables. But the BIC value continues to decrease as I increase the lag (its even the case for k=70 and more which is unrealistic) so the BIC, ...
0
votes
0answers
27 views

Choosing between two econometric models

I created an initial model log(Consumption) = a + b*log(GDP), which showed strong evidence of autocorrelation and heteroscedasticity. I've created two different models to address this but I'm ...
2
votes
2answers
76 views

Is overfitting always a problem?

If I test various models and the best performing model also happens to be one that appears to be overfit, is this an issue? For example, if I have a model with 100% accuracy on the training data and ...
0
votes
0answers
14 views

What is the best test to use?

I would like to design a study that looks at the relationship between perceived competence and perceived helpfulness of website information for two separate companies. Say I asked participants to ...
2
votes
3answers
71 views

Can I guess which time series model (ARIMA, SARIMA) I should use just by looking at the time series plot?

I have the time series plot shown above. Is is possible to know which model I should use solely by looking at this plot?
2
votes
0answers
12 views

Is inference after model selection a real problem when few models are manually compared?

I performed an experiment that went wrong, but the error made it possible to test another hypothesis that I generated upon noticing the error (and before knowing the outcome). Parameter A should have ...
1
vote
0answers
24 views

Prediction of financial time series

I have several questions. I will split the text up in one high-level description of the goal of my exercise, a detailed description of my potential solution and finally my actual questions. Please ...
1
vote
1answer
72 views

Bayes factors from MCMC samples

I'm working to implement Bayesian model selection among models whose posteriors have already been sampled via MCMC. After reviewing some discussions of Bayes factors, I understand that they are ...
0
votes
0answers
25 views

Model selection: Comparing nested models without likelihood

Say I have a set of three models $M = \{M_1, M_2, M_3\}$ that are nested, i.e. $M_1$ is a constrained version of $M_2$ which again is a constrained version of $M_3$. I have a set of data $X$ with some ...
0
votes
0answers
10 views

Econometric model choice after Breusch and Pagan?

I have panel data and as a new Stata user, I have a question regarding my model choice: After performing a Breusch and Pagan test for RE, the result comes out significant indicating that I am to use ...
0
votes
1answer
25 views

How to give more importance to one variable in a logistic regresion model? [closed]

I'm adjusting a logistic regression model for prediction, but if the person interested says: All variables are important for me, but especially X2 is more important. How I give that variable more ...
0
votes
1answer
19 views

Efficient way to estimate the order p for autoregressive model AR(p)?

I am writing an algorithms to build AR model to estimate stock price in the future. However, I have 88 stocks to look at and wonder whether there is any efficient way to estimate the order p for all ...
2
votes
2answers
34 views

What can I consider to choose between the same model but estimated with different estimators?

I estimated a standard regression equation with ML and GMM. The question is: how can I know which estimator provides the best estimate? (e.g., the GMM is more efficient if errors are not normally ...
0
votes
0answers
12 views

Decision tree model selection process

Trying to apply train, validation, and test set to a Binary Decision Tree classifier, following the logic of https://www.coursera.org/learn/machine-learning/home/welcome. The logic is as follows, if ...
2
votes
0answers
17 views

LASSO Cox Model after multiple imputation

I want to develop a predictive survival model on a data set with about 8000 subjects and 38 covariates. About 4% of subjects had the event of interest. There are 21 variables with missing values, ...
0
votes
0answers
12 views

What is the prerequisites “the same dataset” for AIC/BIC?

Let make a example. Suppose I'm doing model selection and my observation data is $Y_{N\times 1}$ and $X_{N\times K}$.(More specify, K=6) Now I have two model, M1 and M2. M1 includes the first ...
1
vote
0answers
36 views

Selecting lag length for VAR Model. Differences or Levels?

I'm currently testing for optimal VAR lag length using the information criteria. I found that my variables are non-stationary (i.e. they have to be first differenced). When I identify the number of ...
2
votes
1answer
36 views

Tuning distance threshold in online clustering

In online clustering there are approaches where a threshold $r$ on the distance to the nearest cluster is used to determine whether a new data point should be associated to an existing cluster or ...
1
vote
0answers
60 views

Determination or AR and MA parameters

I have in my possession price and time of different trade from an auction. The price series isn't stationnary so I work with the log return series. I'd like to forecast the evolution of the log return ...
0
votes
0answers
119 views

Confused with regsubsets() in R

I am a little confused with regsubsets in R, and the different methods- AIC, Cp and Adjusted $R^2$ in general. Suppose our model has $p$ predictors. From the output ...
0
votes
0answers
53 views

How to estimate model predicted means per group from a GEE model fitted in R?

This question is related to a similar post: How can I estimate model predicted means (a.k.a. marginal means, lsmeans, or EM means) from a GEE model fitted in R? The difference is that I do not only ...
0
votes
1answer
33 views

What model to use [closed]

I have data on whether an audition was successful or not and some data that could help explain the success/failure up to some extent (I have about 10 categorical variables and 2 continuous). So the ...
0
votes
0answers
11 views

What is a “subclass of linear models”?

I found the definition of graphical model structure in a paper, and it used the term "a subclass of linear models has graphical model structure if ...". What exactly is a class and a subclass of ...
1
vote
2answers
244 views

Time series Forecasting in Python - comparing different models

I have a large number of different timeseries and I need to create a forecast for each one of them. Are there packages that enable: Auto tuning of models? Cross validation of different tuned models -...
2
votes
1answer
45 views

K-NN optimal value of 'K'

How can I find the optimal value of 'K' in K-NN using the cross_val_score function, with scoring metric as auc_score? Do I need ...
12
votes
4answers
263 views

What is the point of univariate regression before multivariate regression?

I am currently working on a problem in which we have a small dataset and are interested in the causality effect of a treatment on the outcome. My advisor has instructed me to perform a univariate ...
2
votes
2answers
64 views

Which ARIMA model do I have to select?

I have monthly data since 1990 to January 2019 and I need to make a forecast of the next 6 months so I use a sample of the past ten years(I used a sample of the last ten years due to my job ...
3
votes
2answers
116 views

What's the real purpose of cross validation?

As for cross evaluation (CV), I have two questions to ask: 1) CV has nothing to do with parameter selection, but only model evaluation? Specifically, which model? 2) In k-fold CV, what's the final ...
2
votes
0answers
56 views

Model selection with different fixed effects and different corARMA structures

I analyzed the effect of temperature (4 different areas) on laying date: LDT ~ Aa3+Bb+Cc+Dd. Because of autocorrelation in residuals I used ...
1
vote
1answer
41 views

How do you select the best from a number of linear models?

I'm learning linear regression with the Carseats data set. I went through the data, cleaned it, encoded the dummy variables and checked for collinearity. The dataset has 400 observations on Carseat ...
1
vote
1answer
45 views

ABC, compute Bayes factor from posteriors

I am pretty new to ABC stuff so I may be saying dumb things. My question is: I ran an ABC with two models $M_1$ and $M_2$ and now I have an approximation of the posterior distribution for both model. ...
1
vote
0answers
17 views

Average time series forecast errors from cross-validation with rolling origin

I'm calculating the MAPE and RMSE over a rolling origin cross-validation with fixed forecast interval for several models. For example, for a daily series with 3 years, I'm training my model with 2 ...
0
votes
0answers
31 views

Bias correction when using loo cross-validation to replace unreliable PSIS-LOO estimates

The PSIS-LOO information criterion (see this paper by Vehtari, Gelman, and Gabry) assigns a Pareto shape parameter $\hat k$ to each observation in the data, and these $\hat k$ values can be used to ...