Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

Filter by
Sorted by
Tagged with
0 votes
0 answers
23 views

Compare bootstrap auc confidence interval using t-test

In order to choose between a machine learning model when the number of features is 5 and a machine learning model when the number of features is 6, I want to bootstrap the auc of the model to obtain a ...
JAE's user avatar
  • 79
0 votes
0 answers
8 views

Select classification model using nested cv and bootstrap auc confidence interval

My goal is to find the best 1 model out of 55 classification models. I first ran nested cv on 55 models to see which model had better generalization. The AUC score was used as an evaluation indicator. ...
JAE's user avatar
  • 79
3 votes
0 answers
162 views

Minimum Description Length, Normalized Maximum Likelihood, and Maximum A Posteriori Estimation

TL;DR: I believe MDL using NML is a special case of the joint MAP of model and parameters, and need to verify this and find sources that have acknowledges this. This is how I understand Minimum ...
Feri's user avatar
  • 316
0 votes
0 answers
17 views

Overfitting Checking in Cross-validated models for Hyperparameter Tuning and Model Selection Settings

Checking overfitting with cross-validation set metrics for hyperparameter tuning and model selection I have basically considered a set of models and pre-processing steps to tune the hyperparameters ...
Adrià Termes Berruga's user avatar
0 votes
0 answers
18 views

Interpreting Contradictory Results in Bayesian Model Averaging: High Posterior Inclusion Probability with Unclear Effect

In my research, I am utilizing the Bayesian Model Averaging (BMA) methodology to identify the best set of regressors that can predict the outcome variable $y$. My dataset consists of five variables ...
Valerio's user avatar
  • 37
0 votes
0 answers
16 views

Select the most general machine learning model

For example, let's say that model A had an average train auc of 0.82 and a test auc of 0.79 through cross-validation. The difference between the two scores is 0.03. Let's say that model B has a train ...
JAE's user avatar
  • 79
-1 votes
0 answers
39 views

What's the best way to do cross-validation?

Assume that 3-fold CV is used for comparing performance of logistic regression, SVM, and KNN for a 2-class problem. For each fold, the logistic regression provides the following confusion matrices: ...
Marlon Brando's user avatar
0 votes
0 answers
62 views

ACF and PACF plots to estimate SARIMA orders

I have some data (sales of a particular item at a particular grocery store) which exhibits both trend and seasonality. I fit these trend and seasonality components by doing a linear regression of the ...
Steven Gubkin's user avatar
0 votes
0 answers
33 views

I screwed-up model selection but ended-up with a very good model; am I ok?

In a recent experiment, I made an oversight: I divided my data into training and testing sets and conducted cross-validation for model selection and hyperparameter tuning after having applied Boruta (...
Alek Fröhlich's user avatar
1 vote
0 answers
12 views

Model choice based on test/train/validation split [duplicate]

My question is very simple, but no matter where I look it up, it seems that I get another answer. Take a simple classification task. Let's say I trained a kNN, LDA and logistic regression on it for ...
Marlon Brando's user avatar
3 votes
1 answer
41 views

What's the relationship between "bias-variance tradeoff" and "consistent model selection"?

I'm very confused about the relationship between "bias-variance tradeoff" and "consistent model selection". Based on my current interpretation, the ultimate goal of taking care of ...
ExcitedSnail's user avatar
  • 2,778
3 votes
1 answer
190 views

Reduce the model sequentially

I was given an ANOVA table and asked to reduce the model sequentially. I searched the online resources say: When reducing the model sequentially, you typically start by assessing the significance of ...
Matata's user avatar
  • 671
2 votes
0 answers
35 views

Confusion about Mallows' Cp

I am trying to use Mallows' $C_p$ to select linear regression models. I have been reading the excellent text by Cosma Shalizi at https://www.stat.cmu.edu/~cshalizi/TALR/TALR.pdf (page 323 to 327). ...
Frank De Geeter's user avatar
1 vote
1 answer
37 views

Selection of best VARX model using VAR() in R

I have 9 variables (all stationary) grouped into five different datasets (each set has 4 common variables and one different). How can I evaluate which is the best VARX model? I'm using ...
Alfonso's user avatar
  • 11
3 votes
0 answers
37 views

What critical level to use in diagnostic tests for model selection in forecasting?

I have been reading Hyndman & Athanasopoulos "Forecasting: Principles and Practice" (newest edition here) recently, and I noticed something that I regard as a possible inconsistency. On ...
Richard Hardy's user avatar
1 vote
1 answer
47 views

What modeling approach should i use AR, MA, ARMA?

those are my ACF and PACF plots for my time series after two differentiating. I watched a couple of tutorials but I cannot figure out what method I am supposed to use. Also, an interpretation of the ...
antekkalafior's user avatar
0 votes
0 answers
25 views

How to fit a dataset like this, and what's the recommended evaluate metrics for it

the dataset seems like non-linear, is there any recommended way to fit the datatset? since it's a non-linear regression problem, what's the correct way to evaluate the model's prediction? is the MSE ...
Wuuu's user avatar
  • 1
0 votes
0 answers
12 views

Online mixture inference; better alternatives than windowed EM?

I have an online Gaussian mixture estimation problem that I would appreciate some input on. To be more precise, I have a stream of scalar observations $x_1, x_2, \dotsc$ arriving over time which are ...
ummg's user avatar
  • 145
1 vote
1 answer
20 views

Is mutual exclusivity important for an A/B test for an audience selection method?

Say I want to measure whether a set of business rules is better than random at identifying customers most likely to respond to an email. The steps are: Take the entire population of 200 people and ...
djs's user avatar
  • 11
0 votes
0 answers
14 views

Comparing and selecting models, constructing objective function (complexity, prior knowledge on the distribution of parameter values)

We have a set of models that were derived using some fitting routine that optimizes parameter values utilizing $\chi^2$ for a given model. model1 has 100 parameters, model2 has 99 parameters, ... ...
twistfire's user avatar
  • 113
1 vote
1 answer
23 views

Interpreting AIC relative likelihoods ( qpcR::akaike.weights() )

I want to ensure that I am correctly interpreting AIC relative likelihood (RL) scores, specifically those returned by qpcR::akaike.weights$rel.LL. For example, I ...
PhelsumaFL's user avatar
7 votes
3 answers
1k views

Is parsimony crucial for statistical inference?

This question is based on using a regression for statistical inference (not prediction). I have conducted hierarchical (logistic mixed effects) regression. The first model includes the predictors of ...
SilvaC's user avatar
  • 512
1 vote
0 answers
22 views

Finding optimal combination of covariates using cross validation

I have a logistic mixed model (lme4 package in R). I want to assess whether participants scores on the measures 'sumspq', 'sumpdi', and 'sumcaps' significantly affect the difference in performance ...
SilvaC's user avatar
  • 512
4 votes
2 answers
172 views

Generalized additive model: Variable & model selection

I know this type of question has been asked many times before, so I apologize for re-posting about it. I bring it up again because it's been taught in one of my courses of study and I want to make ...
Nate's user avatar
  • 1,529
1 vote
0 answers
51 views

How many lags of independent variables to use

I have a panel dataset (26880 observations) of individual decisions ($y_{it}$, a categorical variable) which depends on signals ($x_{it}$). I am trying to find out how many past signals individuals ...
jasmine's user avatar
  • 367
4 votes
1 answer
57 views

How to choose the best one model from ARIMAX, ARCH/GARCH and VAR?

Now I have 3 models to find what economic factors have an effect on gold price: ARIMAX model ARCH/GARCH model VAR model What is the tool to find the best model? In linear regression, we can ...
Susan's user avatar
  • 41
0 votes
0 answers
19 views

F-statistic and p-values plots with lag

I am currently working on multivariate predictive models and have generated several diagrams to represent F-statistics and p-values for different models across various lags. I would greatly appreciate ...
Creative T's user avatar
7 votes
2 answers
125 views

How can I get a best model? An exploratory LMM

I'd like to inquire about the linear mixed model and its application to my dataset. The dataset comprises a dependent variable (DV) denoted as V, alongside three ...
TKw's user avatar
  • 71
11 votes
7 answers
3k views

Why do we use Linear Models when tree based models often work better than linear models?

In Supervised Machine Learning, and specifically on Kaggle, it is usually seen that tree models often outperform linear models. And even in the tree-based models, it is usually XGBoost that ...
letdatado's user avatar
  • 325
1 vote
1 answer
38 views

Auto arima capital D

I've got time series 3 years long, there is seasonal uplift during December - but it's not so clear. Seasonal test fails. I train model twice without setting any parameter: ...
voncuver's user avatar
3 votes
2 answers
83 views

Variable selection in logistic regression [duplicate]

So I'm trying to make a multivariate logistic regression model in R studio. I'm not sure how to go about this. What seemed to make sense to me was to model every predictor against the response ...
AdmiralMunson's user avatar
1 vote
1 answer
70 views

How can Null model likelihood be higher than Fitted model likelihood

As far as I know, when fitting a GLM, the fitted model should always have a higher likelihood compared to the null model (with only an intercept) for the same training set. When I run a small ...
Kozolovska's user avatar
  • 1,355
4 votes
0 answers
27 views

Why do model selection criteria (xICs, etc) not explicitly incorporate a loss function?

Model Selection and Multimodel Inference by Burnham and Anderson notes that TIC, AIC, AICc and QAICc are based on K-L distance between a given model and true model. Also BIC is in a sense based on ...
Mohan's user avatar
  • 865
0 votes
0 answers
45 views

What kind of series is this? Time series Model Choice

I am trying to model a series (mainly as self-education, as a way of deepening my time series knowledge). The series is percentage growth in GDP per capita for the United States, taken from the World ...
Lindsay's user avatar
0 votes
0 answers
23 views

What exactly is the right approach when trying to find OOS MSE when using linear lasso regression?

This isn't a question where I have a code example to provide. It is more of an informal question about what to do between 2 options. Assume I have some data and my goal is to fit a model using the ...
Donk's user avatar
  • 1
1 vote
0 answers
35 views

Hyperparameter selection after nested cross-validation and making comparisons with DeLong's test

I have already read all the associated questions on the topic but couldn't find a clear answer. I initially split my data into training (80%) and hold-out testing (20%). Then, I am performing nested ...
user22409235's user avatar
2 votes
1 answer
91 views

pdf vs probability vs likelihood [duplicate]

How to compute the log likelihood? Let's take a simple example using a normal distribution and scipy to do the work. Assuming X is the data, and the normal distribution as the model (...
brice rebsamen's user avatar
1 vote
1 answer
43 views

Calculate (quasi) AIC for mixed-effect baseline-category (multinomial logit) model

I am doing a discrete choice experiment where respondents are presented with different patient profiles, and for each profile, respondents need to choose one (out of three) treatment options. An ...
Trang Hien's user avatar
4 votes
1 answer
37 views

Does adding a random intercept for subject address confounding variables within subjects (e.g. sex or age)?

Let's say I am interested in identifying associations between a blood protein and disease activity, but I have multiple measurements per subject. Based on a literature review, I expect sex differences ...
HarD's user avatar
  • 217
1 vote
0 answers
48 views

Specifying parameters for SARIMAX model with significant ACF / PACF at tails

I have hourly data that has a period of 1 day or 24 hours / time steps and I hope to do short term forecasting for a few days in advance. The ACF of the raw time series was periodic (see last figure) ...
Yandle's user avatar
  • 1,169
1 vote
1 answer
41 views

LRT & AIC comparison, selecting a model fitted by weighted LS

I am a completely newbie in statistics and I want to ask a couple of questions about Akaike Information Criteria & Likellihood Ratio Test for particular application. I am trying to fit data using ...
twistfire's user avatar
  • 113
2 votes
1 answer
79 views

R-squared vs adjusted R-squared in Hierarchical multiple regression

In hierarchical multiple regression (not to be confused with hierarchical linear models that account for variance components), you add model terms by block. The fit of the new model is measured by the ...
Migs F's user avatar
  • 21
0 votes
0 answers
35 views

BMA formula with BIC

I am interested in using Bayesian modele averaging as a selection creteria (BMA) vs AIC. I read that BMA is widely implemented in clustering models. Suppose that we need to fit M models to a data and ...
Alice's user avatar
  • 640
0 votes
0 answers
16 views

Model Selection on Coxme

I am building a "time to event" model using the coxme package in R. I have a lot of mixed effects (15) that I want to input into a global model and find the best fit for - is there a model ...
michellemoyah's user avatar
0 votes
0 answers
27 views

Total generalized variance for Box-Cox transformed components

I have a couple Gaussian mixture models where each component comes from (component-wise) Box-Cox transformed data. These models do not describe the same data: the individual components are selected ...
ladislaw94's user avatar
1 vote
1 answer
201 views

Which regression model would you choose?

Which regression model would you choose to model the following flood damage data? The variables are x1=water height, x2=dike height and x3=flood damage. The following plot shows how the flood damages ...
Sjafnargata's user avatar
1 vote
1 answer
34 views

train / validation / test split problem

Suppose that I have created train/validation/test splits for model building. I optimized the hyperparameters using the validation set and chose the parameter values which gave the highest accuracy. To ...
Sanyo Mn's user avatar
  • 1,252
2 votes
1 answer
76 views

Nested models: should we use criteria like adjusted R squared, AIC/BIC, Mallows' Cp or the F-test?

I am confused about which one to use for, say testing $ H_0: \beta_1 = \beta_2 $. The F-test for comparison of residual sum of squares can be used here, as can things like adjusted $R^2$, AIC/BIC or ...
Ariel4778's user avatar
0 votes
0 answers
22 views

Is there any bias introduced by evaluating a model and decisions based on this model on the same data set?

As an example, let's say we have some financial time series such as closing prices of some stock and we would like to evaluate the ability of different models to forecast future closing prices as well ...
QMath's user avatar
  • 389
2 votes
1 answer
46 views

Model Building Process Without Feature Names/Domain Knowledge

I am working with a dataset of $\approx 10^7$ samples and $\approx 120$ features. All samples have a binary classification $0 - 1$. I am attempting to build a model that minimizes out-of-sample error ...
Bepop's user avatar
  • 307

1
2 3 4 5
40