# Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

2,003 questions
Filter by
Sorted by
Tagged with
1 vote
12 views

### How to interpret model fit in posterior predictive checks between two models that both capture the observations in its 1sigma?

I have two models aimed at explaining a single observed measurement $x_{obs}$: Simple Model with 26 parameters $f_1(\theta)$. Complex Model with 31 parameters $f_2(\theta)$. Both models are assumed ...
13 views

### Hierarchical models where the hierarchy structure depends on a latent variable

I am having trouble formulating a hierarchical model for the purpose of Bayesian inference in the case where the actual hierarchical structure depends on a latent variable. I am wondering if this is ...
• 131
143 views

• 2,711
13 views

### Inference for a data-adaptive parameter

You have data on a continuous variable $A$, representing the dosage of a drug, and a continuous outcome $Y$, representing some health outcome. I want to estimate the "optimal" dosage of $A$, ...
• 1,182
12 views

### Can I use voting in nested cross validation to obtain estimator?

NEWBIE ALERT: First post, so be gentle :-) I have looked at nested cross validation (as e.g. in this Medium article). I understand that I am able to obtain averaged scores from the outer loop and that ...
106 views

### How compute a BIC for multivariate regression?

I would like to know if it is possible to compute a BIC for a multivariate regression (One predictor X and 3 responses outcomes Y). If yes, how? In R, when I run: ...
• 41
65 views

### Confused on Bayesian Decision Theory

I am trying to understand what is the right way to pick up an "action", as it is called in Murphy, Machine Learning a Probabilistic Perspective, in the 'chatper 'Bayesian decision theory'. I'...
14 views

### CPO, DIC or WAIC, which metric to choose when they don't agree?

I am creating a Bayesian spatiotemporal model with the four type Knorr Held interaction proposal. I am trying the different type of interactions and I want to select the best model based on DIC, WAIC ...
• 23
39 views

### Questions regarding the definition of the deviance in the context of GLMs

I've been self-studying GLMs and I have some questions regarding the deviance in the context of GLMs. In Generalized Additive Models An Introduction with R, the author defines the deviance of a model ...
• 21
28 views

### Interpret the PACF plot to select the correct lag (AR model order)

I want to select lag (AR model order) for the series Food price inflation. AIC gives 4. SIC gives 3. And also, I print its PACF plot. How can I interpret the PACF plot to select the correct lag?
• 1,132
24 views

### Higher order moments to evaluate strength of linear relationship between variables

Let $X_1,\dots,X_n$ be real random variables such that $\alpha_1X_1+\dots+\alpha_nX_n=0$ for some unknown $\alpha_1,\dots,\alpha_n$. If $n=2$, one can study the strength of linear relationship by ...
• 213
104 views

### Any Insights on the adoption and use of the Healthy Akaike Information Criterion (hAIC)?

Recently, I came across the Healthy Akaike Information Criterion (hAIC), introduced by Demidenko in his 2004 book "Mixed Models: Theory and Applications with R." Despite its (potential) ...
• 63.6k
15 views

### How to split data when training and tuning the meta learner in stacking?

I have a simple yet tricky conceptual question about the data splitting of a meta learning process. Assume I have a simple X_train, ...
• 53
130 views

### Choosing Between Intercept-Only and AR-NN Models: Justified to not use the model with the lowest RMSE/MAE?

I have created two autoregressive models for forecasting: a basic intercept-only model and an AR-NN (autoregressive neural network) model. Both models show similar performance based on recursive one-...
• 361
8 views

### Can you deduce if a lasso model has a smaller/larger/equal RSS to a forward selection model?

I came across this question in my exam. Where there is a table where the columns are the different model selection methods: OLS, Lasso, Forward_Size1, ForwardSize2. And the rows are the predictors, ...
1 vote
44 views

• 187
46 views

### Compare bootstrap auc confidence interval using t-test

In order to choose between a machine learning model when the number of features is 5 and a machine learning model when the number of features is 6, I want to bootstrap the auc of the model to obtain a ...
• 79
13 views

### Select classification model using nested cv and bootstrap auc confidence interval

My goal is to find the best 1 model out of 55 classification models. I first ran nested cv on 55 models to see which model had better generalization. The AUC score was used as an evaluation indicator. ...
• 79
288 views

### Minimum Description Length, Normalized Maximum Likelihood, and Maximum A Posteriori Estimation

TL;DR: I believe MDL using NML is a special case of the joint MAP of model and parameters, and need to verify this and find sources that have acknowledges this. This is how I understand Minimum ...
• 187
22 views

### Interpreting Contradictory Results in Bayesian Model Averaging: High Posterior Inclusion Probability with Unclear Effect

In my research, I am utilizing the Bayesian Model Averaging (BMA) methodology to identify the best set of regressors that can predict the outcome variable $y$. My dataset consists of five variables ...
• 37
16 views

### Select the most general machine learning model

For example, let's say that model A had an average train auc of 0.82 and a test auc of 0.79 through cross-validation. The difference between the two scores is 0.03. Let's say that model B has a train ...
• 79
68 views

### ACF and PACF plots to estimate SARIMA orders

I have some data (sales of a particular item at a particular grocery store) which exhibits both trend and seasonality. I fit these trend and seasonality components by doing a linear regression of the ...
33 views

### I screwed-up model selection but ended-up with a very good model; am I ok?

In a recent experiment, I made an oversight: I divided my data into training and testing sets and conducted cross-validation for model selection and hyperparameter tuning after having applied Boruta (...
1 vote
14 views

### Model choice based on test/train/validation split [duplicate]

My question is very simple, but no matter where I look it up, it seems that I get another answer. Take a simple classification task. Let's say I trained a kNN, LDA and logistic regression on it for ...
61 views

### What's the relationship between "bias-variance tradeoff" and "consistent model selection"?

I'm very confused about the relationship between "bias-variance tradeoff" and "consistent model selection". Based on my current interpretation, the ultimate goal of taking care of ...
• 2,956
194 views

### Reduce the model sequentially

I was given an ANOVA table and asked to reduce the model sequentially. I searched the online resources say: When reducing the model sequentially, you typically start by assessing the significance of ...
• 671
50 views

I am trying to use Mallows' $C_p$ to select linear regression models. I have been reading the excellent text by Cosma Shalizi at https://www.stat.cmu.edu/~cshalizi/TALR/TALR.pdf (page 323 to 327). ...
1 vote
111 views

### Selection of best VARX model using VAR() in R

I have 9 variables (all stationary) grouped into five different datasets (each set has 4 common variables and one different). How can I evaluate which is the best VARX model? I'm using ...
• 21
38 views

### What critical level to use in diagnostic tests for model selection in forecasting?

I have been reading Hyndman & Athanasopoulos "Forecasting: Principles and Practice" (newest edition here) recently, and I noticed something that I regard as a possible inconsistency. On ...
• 68.8k
1 vote
48 views

### What modeling approach should i use AR, MA, ARMA?

those are my ACF and PACF plots for my time series after two differentiating. I watched a couple of tutorials but I cannot figure out what method I am supposed to use. Also, an interpretation of the ...
26 views

### How to fit a dataset like this, and what's the recommended evaluate metrics for it

the dataset seems like non-linear, is there any recommended way to fit the datatset? since it's a non-linear regression problem, what's the correct way to evaluate the model's prediction? is the MSE ...
• 1
15 views

### Online mixture inference; better alternatives than windowed EM?

I have an online Gaussian mixture estimation problem that I would appreciate some input on. To be more precise, I have a stream of scalar observations $x_1, x_2, \dotsc$ arriving over time which are ...
• 145
1 vote
23 views

### Is mutual exclusivity important for an A/B test for an audience selection method?

Say I want to measure whether a set of business rules is better than random at identifying customers most likely to respond to an email. The steps are: Take the entire population of 200 people and ...
• 11
16 views

### Comparing and selecting models, constructing objective function (complexity, prior knowledge on the distribution of parameter values)

We have a set of models that were derived using some fitting routine that optimizes parameter values utilizing $\chi^2$ for a given model. model1 has 100 parameters, model2 has 99 parameters, ... ...
• 113
31 views

• 357
83 views

### How to choose the best one model from ARIMAX, ARCH/GARCH and VAR?

Now I have 3 models to find what economic factors have an effect on gold price: ARIMAX model ARCH/GARCH model VAR model What is the tool to find the best model? In linear regression, we can ...
• 31
24 views

### F-statistic and p-values plots with lag

I am currently working on multivariate predictive models and have generated several diagrams to represent F-statistics and p-values for different models across various lags. I would greatly appreciate ...
145 views

### How can I get a best model? An exploratory LMM

I'd like to inquire about the linear mixed model and its application to my dataset. The dataset comprises a dependent variable (DV) denoted as V, alongside three ...
• 71
3k views

### Why do we use Linear Models when tree based models often work better than linear models?

In Supervised Machine Learning, and specifically on Kaggle, it is usually seen that tree models often outperform linear models. And even in the tree-based models, it is usually XGBoost that ...
• 357
1 vote
44 views

### Auto arima capital D

I've got time series 3 years long, there is seasonal uplift during December - but it's not so clear. Seasonal test fails. I train model twice without setting any parameter: ...
• 25
204 views

### Variable selection in logistic regression [duplicate]

So I'm trying to make a multivariate logistic regression model in R studio. I'm not sure how to go about this. What seemed to make sense to me was to model every predictor against the response ...
1 vote
80 views

### How can Null model likelihood be higher than Fitted model likelihood

As far as I know, when fitting a GLM, the fitted model should always have a higher likelihood compared to the null model (with only an intercept) for the same training set. When I run a small ...
• 1,365