Questions tagged [model-selection]
Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.
1,994
questions
0
votes
0
answers
17
views
What's the best way to do cross-validation? [closed]
Assume we make a threefold cross validation in order to compare logistic regression, SVM and KNN. Now, for the Logistic Regression it turns out:
Block as Validation Set
Correct 5; Wrong 1
Block as ...
0
votes
0
answers
39
views
+100
ACF and PACF plots to estimate SARIMA orders
I have some data (sales of a particular item at a particular grocery store) which exhibits both trend and seasonality. I fit these trend and seasonality components by doing a linear regression of the ...
0
votes
0
answers
30
views
I screwed-up model selection but ended-up with a very good model; am I ok?
In a recent experiment, I made an oversight: I divided my data into training and testing sets and conducted cross-validation for model selection and hyperparameter tuning after having applied Boruta (...
1
vote
0
answers
12
views
Model choice based on test/train/validation split [duplicate]
My question is very simple, but no matter where I look it up, it seems that I get another answer.
Take a simple classification task. Let's say I trained a kNN, LDA and logistic regression on it for ...
2
votes
1
answer
33
views
What's the relationship between "bias-variance tradeoff" and "consistent model selection"?
I'm very confused about the relationship between "bias-variance tradeoff" and "consistent model selection". Based on my current interpretation, the ultimate goal of taking care of ...
3
votes
1
answer
186
views
Reduce the model sequentially
I was given an ANOVA table and asked to reduce the model sequentially.
I searched the online resources say: When reducing the model sequentially, you typically start by assessing the significance of ...
2
votes
0
answers
32
views
Confusion about Mallows' Cp
I am trying to use Mallows' $C_p$ to select linear regression models. I have been reading the excellent text by Cosma Shalizi at https://www.stat.cmu.edu/~cshalizi/TALR/TALR.pdf
(page 323 to 327).
...
1
vote
1
answer
29
views
Selection of best VARX model using VAR() in R
I have 9 variables (all stationary) grouped into five different datasets (each set has 4 common variables and one different). How can I evaluate which is the best VARX model? I'm using ...
3
votes
0
answers
33
views
What critical level to use in diagnostic tests for model selection in forecasting?
I have been reading Hyndman & Athanasopoulos "Forecasting: Principles and Practice" (newest edition here) recently, and I noticed something that I regard as a possible inconsistency. On ...
1
vote
1
answer
40
views
What modeling approach should i use AR, MA, ARMA?
those are my ACF and PACF plots for my time series after two differentiating. I watched a couple of tutorials but I cannot figure out what method I am supposed to use. Also, an interpretation of the ...
0
votes
0
answers
25
views
How to fit a dataset like this, and what's the recommended evaluate metrics for it
the dataset seems like non-linear,
is there any recommended way to fit the datatset? since it's a non-linear regression problem, what's the correct way to evaluate the model's prediction? is the MSE ...
0
votes
0
answers
12
views
Online mixture inference; better alternatives than windowed EM?
I have an online Gaussian mixture estimation problem that I would appreciate some input on. To be more precise, I have a stream of scalar observations $x_1, x_2, \dotsc$ arriving over time which are ...
1
vote
1
answer
20
views
Is mutual exclusivity important for an A/B test for an audience selection method?
Say I want to measure whether a set of business rules is better than random at identifying customers most likely to respond to an email. The steps are:
Take the entire population of 200 people and ...
0
votes
0
answers
12
views
Comparing and selecting models, constructing objective function (complexity, prior knowledge on the distribution of parameter values)
We have a set of models that were derived using some fitting routine that optimizes parameter values utilizing $\chi^2$ for a given model.
model1 has 100 parameters,
model2 has 99 parameters,
...
...
1
vote
1
answer
21
views
Interpreting AIC relative likelihoods ( qpcR::akaike.weights() )
I want to ensure that I am correctly interpreting AIC relative likelihood (RL) scores, specifically those returned by qpcR::akaike.weights$rel.LL. For example, I ...
7
votes
3
answers
1k
views
Is parsimony crucial for statistical inference?
This question is based on using a regression for statistical inference (not prediction).
I have conducted hierarchical (logistic mixed effects) regression.
The first model includes the predictors of ...
1
vote
0
answers
22
views
Finding optimal combination of covariates using cross validation
I have a logistic mixed model (lme4 package in R). I want to assess whether participants scores on the measures 'sumspq', 'sumpdi', and 'sumcaps' significantly affect the difference in performance ...
4
votes
2
answers
149
views
Generalized additive model: Variable & model selection
I know this type of question has been asked many times before, so I apologize for re-posting about it. I bring it up again because it's been taught in one of my courses of study and I want to make ...
1
vote
0
answers
49
views
How many lags of independent variables to use
I have a panel dataset (26880 observations) of individual decisions ($y_{it}$, a categorical variable) which depends on signals ($x_{it}$). I am trying to find out how many past signals individuals ...
4
votes
1
answer
55
views
How to choose the best one model from ARIMAX, ARCH/GARCH and VAR?
Now I have 3 models to find what economic factors have an effect on gold price:
ARIMAX model
ARCH/GARCH model
VAR model
What is the tool to find the best model? In linear regression, we can ...
0
votes
0
answers
17
views
F-statistic and p-values plots with lag
I am currently working on multivariate predictive models and have generated several diagrams to represent F-statistics and p-values for different models across various lags. I would greatly appreciate ...
7
votes
2
answers
121
views
How can I get a best model? An exploratory LMM
I'd like to inquire about the linear mixed model and its application to my dataset. The dataset comprises a dependent variable (DV) denoted as V, alongside three ...
11
votes
7
answers
2k
views
Why do we use Linear Models when tree based models often work better than linear models?
In Supervised Machine Learning, and specifically on Kaggle, it is usually seen that tree models often outperform linear models. And even in the tree-based models, it is usually XGBoost that ...
1
vote
1
answer
35
views
Auto arima capital D
I've got time series 3 years long, there is seasonal uplift during December - but it's not so clear. Seasonal test fails.
I train model twice without setting any parameter:
...
3
votes
2
answers
72
views
Variable selection in logistic regression [duplicate]
So I'm trying to make a multivariate logistic regression model in R studio. I'm not sure how to go about this. What seemed to make sense to me was to model every predictor against the response ...
1
vote
1
answer
70
views
How can Null model likelihood be higher than Fitted model likelihood
As far as I know, when fitting a GLM, the fitted model should always have a higher likelihood compared to the null model (with only an intercept) for the same training set. When I run a small ...
4
votes
0
answers
26
views
Why do model selection criteria (xICs, etc) not explicitly incorporate a loss function?
Model Selection and Multimodel Inference by Burnham and Anderson notes that TIC, AIC, AICc and QAICc are based on K-L distance between a given model and true model. Also BIC is in a sense based on ...
0
votes
0
answers
45
views
What kind of series is this? Time series Model Choice
I am trying to model a series (mainly as self-education, as a way of deepening my time series knowledge). The series is percentage growth in GDP per capita for the United States, taken from the World ...
0
votes
0
answers
23
views
What exactly is the right approach when trying to find OOS MSE when using linear lasso regression?
This isn't a question where I have a code example to provide. It is more of an informal question about what to do between 2 options.
Assume I have some data and my goal is to fit a model using the ...
1
vote
0
answers
33
views
Hyperparameter selection after nested cross-validation and making comparisons with DeLong's test
I have already read all the associated questions on the topic but couldn't find a clear answer. I initially split my data into training (80%) and hold-out testing (20%). Then, I am performing nested ...
2
votes
1
answer
85
views
pdf vs probability vs likelihood [duplicate]
How to compute the log likelihood?
Let's take a simple example using a normal distribution and scipy to do the work. Assuming X is the data, and the normal distribution as the model (...
1
vote
1
answer
41
views
Calculate (quasi) AIC for mixed-effect baseline-category (multinomial logit) model
I am doing a discrete choice experiment where respondents are presented with different patient profiles, and for each profile, respondents need to choose one (out of three) treatment options. An ...
4
votes
1
answer
33
views
Does adding a random intercept for subject address confounding variables within subjects (e.g. sex or age)?
Let's say I am interested in identifying associations between a blood protein and disease activity, but I have multiple measurements per subject. Based on a literature review, I expect sex differences ...
1
vote
0
answers
45
views
Specifying parameters for SARIMAX model with significant ACF / PACF at tails
I have hourly data that has a period of 1 day or 24 hours / time steps and I hope to do short term forecasting for a few days in advance. The ACF of the raw time series was periodic (see last figure) ...
1
vote
1
answer
40
views
LRT & AIC comparison, selecting a model fitted by weighted LS
I am a completely newbie in statistics and I want to ask a couple of questions about Akaike Information Criteria & Likellihood Ratio Test for particular application.
I am trying to fit data using ...
2
votes
1
answer
71
views
R-squared vs adjusted R-squared in Hierarchical multiple regression
In hierarchical multiple regression (not to be confused with hierarchical linear models that account for variance components), you add model terms by block. The fit of the new model is measured by the ...
0
votes
0
answers
34
views
BMA formula with BIC
I am interested in using Bayesian modele averaging as a selection creteria (BMA) vs AIC. I read that BMA is widely implemented in clustering models.
Suppose that we need to fit M models to a data and ...
0
votes
0
answers
16
views
Model Selection on Coxme
I am building a "time to event" model using the coxme package in R. I have a lot of mixed effects (15) that I want to input into a global model and find the best fit for - is there a model ...
0
votes
0
answers
26
views
Total generalized variance for Box-Cox transformed components
I have a couple Gaussian mixture models where each component comes from (component-wise) Box-Cox transformed data. These models do not describe the same data: the individual components are selected ...
1
vote
1
answer
200
views
Which regression model would you choose?
Which regression model would you choose to model the following flood damage data? The variables are x1=water height, x2=dike height and x3=flood damage. The following plot shows how the flood damages ...
1
vote
1
answer
34
views
train / validation / test split problem
Suppose that I have created train/validation/test splits for model building.
I optimized the hyperparameters using the validation set and chose the parameter values which gave the highest accuracy. To ...
2
votes
1
answer
69
views
Nested models: should we use criteria like adjusted R squared, AIC/BIC, Mallows' Cp or the F-test?
I am confused about which one to use for, say testing $ H_0: \beta_1 = \beta_2 $. The F-test for comparison of residual sum of squares can be used here, as can things like adjusted $R^2$, AIC/BIC or ...
0
votes
0
answers
22
views
Is there any bias introduced by evaluating a model and decisions based on this model on the same data set?
As an example, let's say we have some financial time series such as closing prices of some stock and we would like to evaluate the ability of different models to forecast future closing prices as well ...
2
votes
1
answer
46
views
Model Building Process Without Feature Names/Domain Knowledge
I am working with a dataset of $\approx 10^7$ samples and $\approx 120$ features. All samples have a binary classification $0 - 1$. I am attempting to build a model that minimizes out-of-sample error ...
0
votes
0
answers
17
views
Reporting policy effect for specific subpopulation - how to interract?
I am seeking guidance on model selection when reporting a causal policy effect for a specific subgroup of the panel. I am open to any form of help: paper, maths, intuitions... Below my setup, please ...
1
vote
0
answers
29
views
Best Strategy for Model Training & Selection (Spoiler: Should I Re-Train?)
After a discussion with some colleagues, I've realized we've different views on which is the go-to strategy for model training.
Strategy A: Train-Validation-Test Split and Final Model Selection
...
4
votes
0
answers
31
views
Can cross-validation be involved in model-building rather than validation?
I have a general idea in mind that would go like this:
randomly split the data into training/testing
build a model on the training data by choosing from among candidate predictors
evaluate it on the ...
0
votes
0
answers
25
views
How do you use ordinal response data with several random effects to do robust hypothesis testing?
I want to explore the roles of lung presence and habitat on tadpoles' ability to tolerate low oxygen levels. My experimental design generated several measurements of "responsiveness" (on an ...
0
votes
0
answers
65
views
Tiny partial effects but high R² in GAMs - and the other way around
I am currently running GAMs with mgcv package and am trying to find a good one by looking at summary() and the visual outputs ...
1
vote
1
answer
16
views
Associating a constant value over time with predictor variables
Question: How do you model a static response variable (change between baseline and final timepoint) with a variable that changes over time?
Background: A person's inflammation levels are measured at 0,...