Questions tagged [model-selection]
Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.
2,003
questions
1
vote
0
answers
12
views
How to interpret model fit in posterior predictive checks between two models that both capture the observations in its 1sigma?
I have two models aimed at explaining a single observed measurement $x_{obs}$:
Simple Model with 26 parameters $f_1(\theta)$.
Complex Model with 31 parameters $f_2(\theta)$.
Both models are assumed ...
2
votes
0
answers
13
views
Hierarchical models where the hierarchy structure depends on a latent variable
I am having trouble formulating a hierarchical model for the purpose of Bayesian inference in the case where the actual hierarchical structure depends on a latent variable. I am wondering if this is ...
- 131
4
votes
1
answer
143
views
Effect of sample size on BIC
The BIC is given by
$$BIC = k\ln(n)-2\ln(\hat{L}).$$
Let's say I have a Gaussian model to which I'm fitting a dataset-- pretty typical stuff. The log-likelihood for the Gaussian model is given by
$$\...
- 41
0
votes
0
answers
18
views
Can the derivation of the BIC be simply extended to multivariate observations?
Under the assumption that observations are univariate and i.i.d., the classical definition of the Bayesian Information Criterion for a model $\mathcal{M}$ and a dataset $\mathcal{D}$ is
$$
BIC = -2 \...
- 2,711
0
votes
0
answers
13
views
Inference for a data-adaptive parameter
You have data on a continuous variable $A$, representing the dosage of a drug, and a continuous outcome $Y$, representing some health outcome.
I want to estimate the "optimal" dosage of $A$, ...
- 1,182
0
votes
0
answers
12
views
Can I use voting in nested cross validation to obtain estimator?
NEWBIE ALERT: First post, so be gentle :-)
I have looked at nested cross validation (as e.g. in this Medium article). I understand that I am able to obtain averaged scores from the outer loop and that ...
3
votes
1
answer
106
views
How compute a BIC for multivariate regression?
I would like to know if it is possible to compute a BIC for a multivariate regression (One predictor X and 3 responses outcomes Y). If yes, how?
In R, when I run:
...
- 41
0
votes
1
answer
65
views
Confused on Bayesian Decision Theory
I am trying to understand what is the right way to pick up an "action", as it is called in Murphy, Machine Learning a Probabilistic Perspective, in the 'chatper 'Bayesian decision theory'.
I'...
- 3
0
votes
0
answers
14
views
CPO, DIC or WAIC, which metric to choose when they don't agree?
I am creating a Bayesian spatiotemporal model with the four type Knorr Held interaction proposal. I am trying the different type of interactions and I want to select the best model based on DIC, WAIC ...
- 23
0
votes
0
answers
39
views
Questions regarding the definition of the deviance in the context of GLMs
I've been self-studying GLMs and I have some questions regarding the deviance in the context of GLMs. In Generalized Additive Models An Introduction with R, the author defines the deviance of a model ...
- 21
0
votes
0
answers
28
views
Interpret the PACF plot to select the correct lag (AR model order)
I want to select lag (AR model order) for the series Food price inflation.
AIC gives 4.
SIC gives 3.
And also, I print its PACF plot.
How can I interpret the PACF plot to select the correct lag?
- 1,132
0
votes
0
answers
24
views
Higher order moments to evaluate strength of linear relationship between variables
Let $X_1,\dots,X_n$ be real random variables such that $\alpha_1X_1+\dots+\alpha_nX_n=0$ for some unknown $\alpha_1,\dots,\alpha_n$. If $n=2$, one can study the strength of linear relationship by ...
- 213
9
votes
0
answers
104
views
Any Insights on the adoption and use of the Healthy Akaike Information Criterion (hAIC)?
Recently, I came across the Healthy Akaike Information Criterion (hAIC), introduced by Demidenko in his 2004 book "Mixed Models: Theory and Applications with R." Despite its (potential) ...
- 63.6k
0
votes
0
answers
15
views
How to split data when training and tuning the meta learner in stacking?
I have a simple yet tricky conceptual question about the data splitting of a meta learning process.
Assume I have a simple X_train, ...
- 53
4
votes
1
answer
130
views
Choosing Between Intercept-Only and AR-NN Models: Justified to not use the model with the lowest RMSE/MAE?
I have created two autoregressive models for forecasting: a basic intercept-only model and an AR-NN (autoregressive neural network) model. Both models show similar performance based on recursive one-...
- 361
0
votes
0
answers
8
views
Can you deduce if a lasso model has a smaller/larger/equal RSS to a forward selection model?
I came across this question in my exam. Where there is a table where the columns are the different model selection methods: OLS, Lasso, Forward_Size1, ForwardSize2. And the rows are the predictors, ...
1
vote
1
answer
44
views
BIC with non-negligible priors
I want to do model selection based on the best-fit/MAP/marginal posterior I find from an MCMC and likelihood maximization. I have a likelihood $\mathcal{L}(X|\theta)$, some informative priors $\pi(\...
- 13
1
vote
1
answer
52
views
Estimate number of covariates in Cox regression model
My doubt about overfitting is almost general, but in this particular case is all about survival models. I am working in a case-cohort study, estimating the HR in a cohort where heart attack correspond ...
- 275
10
votes
1
answer
412
views
Bayesian Justification of Cross-validation
If I understand correctly, K-fold cross-validation is supposed to approximate expected log predictive density (ELPD), which is defined as $\mathop{\mathbb{E}}_{D_{new}\sim P(.|M_{true})}\log P(D_{new}|...
- 187
0
votes
0
answers
46
views
Compare bootstrap auc confidence interval using t-test
In order to choose between a machine learning model when the number of features is 5 and a machine learning model when the number of features is 6, I want to bootstrap the auc of the model to obtain a ...
- 79
0
votes
0
answers
13
views
Select classification model using nested cv and bootstrap auc confidence interval
My goal is to find the best 1 model out of 55 classification models.
I first ran nested cv on 55 models to see which model had better generalization. The AUC score was used as an evaluation indicator.
...
- 79
3
votes
0
answers
288
views
Minimum Description Length, Normalized Maximum Likelihood, and Maximum A Posteriori Estimation
TL;DR: I believe MDL using NML is a special case of the joint MAP of model and parameters, and need to verify this and find sources that have acknowledges this.
This is how I understand Minimum ...
- 187
0
votes
0
answers
22
views
Interpreting Contradictory Results in Bayesian Model Averaging: High Posterior Inclusion Probability with Unclear Effect
In my research, I am utilizing the Bayesian Model Averaging (BMA) methodology to identify the best set of regressors that can predict the outcome variable $y$. My dataset consists of five variables ...
- 37
0
votes
0
answers
16
views
Select the most general machine learning model
For example, let's say that model A had an average train auc of 0.82 and a test auc of 0.79 through cross-validation. The difference between the two scores is 0.03.
Let's say that model B has a train ...
- 79
0
votes
0
answers
68
views
ACF and PACF plots to estimate SARIMA orders
I have some data (sales of a particular item at a particular grocery store) which exhibits both trend and seasonality. I fit these trend and seasonality components by doing a linear regression of the ...
- 555
0
votes
0
answers
33
views
I screwed-up model selection but ended-up with a very good model; am I ok?
In a recent experiment, I made an oversight: I divided my data into training and testing sets and conducted cross-validation for model selection and hyperparameter tuning after having applied Boruta (...
- 151
1
vote
0
answers
14
views
Model choice based on test/train/validation split [duplicate]
My question is very simple, but no matter where I look it up, it seems that I get another answer.
Take a simple classification task. Let's say I trained a kNN, LDA and logistic regression on it for ...
- 623
3
votes
1
answer
61
views
What's the relationship between "bias-variance tradeoff" and "consistent model selection"?
I'm very confused about the relationship between "bias-variance tradeoff" and "consistent model selection". Based on my current interpretation, the ultimate goal of taking care of ...
- 2,956
3
votes
1
answer
194
views
Reduce the model sequentially
I was given an ANOVA table and asked to reduce the model sequentially.
I searched the online resources say: When reducing the model sequentially, you typically start by assessing the significance of ...
- 671
2
votes
0
answers
50
views
Confusion about Mallows' Cp
I am trying to use Mallows' $C_p$ to select linear regression models. I have been reading the excellent text by Cosma Shalizi at https://www.stat.cmu.edu/~cshalizi/TALR/TALR.pdf
(page 323 to 327).
...
1
vote
1
answer
111
views
Selection of best VARX model using VAR() in R
I have 9 variables (all stationary) grouped into five different datasets (each set has 4 common variables and one different). How can I evaluate which is the best VARX model? I'm using ...
- 21
2
votes
0
answers
38
views
What critical level to use in diagnostic tests for model selection in forecasting?
I have been reading Hyndman & Athanasopoulos "Forecasting: Principles and Practice" (newest edition here) recently, and I noticed something that I regard as a possible inconsistency. On ...
- 68.8k
1
vote
1
answer
48
views
What modeling approach should i use AR, MA, ARMA?
those are my ACF and PACF plots for my time series after two differentiating. I watched a couple of tutorials but I cannot figure out what method I am supposed to use. Also, an interpretation of the ...
- 143
0
votes
0
answers
26
views
How to fit a dataset like this, and what's the recommended evaluate metrics for it
the dataset seems like non-linear,
is there any recommended way to fit the datatset? since it's a non-linear regression problem, what's the correct way to evaluate the model's prediction? is the MSE ...
- 1
0
votes
0
answers
15
views
Online mixture inference; better alternatives than windowed EM?
I have an online Gaussian mixture estimation problem that I would appreciate some input on. To be more precise, I have a stream of scalar observations $x_1, x_2, \dotsc$ arriving over time which are ...
- 145
1
vote
1
answer
23
views
Is mutual exclusivity important for an A/B test for an audience selection method?
Say I want to measure whether a set of business rules is better than random at identifying customers most likely to respond to an email. The steps are:
Take the entire population of 200 people and ...
- 11
0
votes
0
answers
16
views
Comparing and selecting models, constructing objective function (complexity, prior knowledge on the distribution of parameter values)
We have a set of models that were derived using some fitting routine that optimizes parameter values utilizing $\chi^2$ for a given model.
model1 has 100 parameters,
model2 has 99 parameters,
...
...
- 113
0
votes
1
answer
31
views
Interpreting AIC relative likelihoods ( qpcR::akaike.weights() )
I want to ensure that I am correctly interpreting AIC relative likelihood (RL) scores, specifically those returned by qpcR::akaike.weights$rel.LL. For example, I ...
- 61
7
votes
3
answers
1k
views
Is parsimony crucial for statistical inference?
This question is based on using a regression for statistical inference (not prediction).
I have conducted hierarchical (logistic mixed effects) regression.
The first model includes the predictors of ...
- 512
1
vote
0
answers
24
views
Finding optimal combination of covariates using cross validation
I have a logistic mixed model (lme4 package in R). I want to assess whether participants scores on the measures 'sumspq', 'sumpdi', and 'sumcaps' significantly affect the difference in performance ...
- 512
4
votes
2
answers
316
views
Generalized additive model: Variable & model selection
I know this type of question has been asked many times before, so I apologize for re-posting about it. I bring it up again because it's been taught in one of my courses of study and I want to make ...
- 1,673
1
vote
0
answers
60
views
How many lags of independent variables to use
I have a panel dataset (26880 observations) of individual decisions ($y_{it}$, a categorical variable) which depends on signals ($x_{it}$). I am trying to find out how many past signals individuals ...
- 357
3
votes
1
answer
83
views
How to choose the best one model from ARIMAX, ARCH/GARCH and VAR?
Now I have 3 models to find what economic factors have an effect on gold price:
ARIMAX model
ARCH/GARCH model
VAR model
What is the tool to find the best model? In linear regression, we can ...
- 31
0
votes
0
answers
24
views
F-statistic and p-values plots with lag
I am currently working on multivariate predictive models and have generated several diagrams to represent F-statistics and p-values for different models across various lags. I would greatly appreciate ...
- 31
7
votes
2
answers
145
views
How can I get a best model? An exploratory LMM
I'd like to inquire about the linear mixed model and its application to my dataset. The dataset comprises a dependent variable (DV) denoted as V, alongside three ...
- 71
11
votes
7
answers
3k
views
Why do we use Linear Models when tree based models often work better than linear models?
In Supervised Machine Learning, and specifically on Kaggle, it is usually seen that tree models often outperform linear models. And even in the tree-based models, it is usually XGBoost that ...
- 357
1
vote
1
answer
44
views
Auto arima capital D
I've got time series 3 years long, there is seasonal uplift during December - but it's not so clear. Seasonal test fails.
I train model twice without setting any parameter:
...
- 25
3
votes
2
answers
204
views
Variable selection in logistic regression [duplicate]
So I'm trying to make a multivariate logistic regression model in R studio. I'm not sure how to go about this. What seemed to make sense to me was to model every predictor against the response ...
- 127
1
vote
1
answer
80
views
How can Null model likelihood be higher than Fitted model likelihood
As far as I know, when fitting a GLM, the fitted model should always have a higher likelihood compared to the null model (with only an intercept) for the same training set. When I run a small ...
- 1,365
4
votes
0
answers
27
views
Why do model selection criteria (xICs, etc) not explicitly incorporate a loss function?
Model Selection and Multimodel Inference by Burnham and Anderson notes that TIC, AIC, AICc and QAICc are based on K-L distance between a given model and true model. Also BIC is in a sense based on ...
- 929