Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

What's the best way to do cross-validation? [closed]

Assume we make a threefold cross validation in order to compare logistic regression, SVM and KNN. Now, for the Logistic Regression it turns out: Block as Validation Set Correct 5; Wrong 1 Block as ...
Marlon Brando's user avatar
0 votes
0 answers
39 views
+100

ACF and PACF plots to estimate SARIMA orders

I have some data (sales of a particular item at a particular grocery store) which exhibits both trend and seasonality. I fit these trend and seasonality components by doing a linear regression of the ...
Steven Gubkin's user avatar
0 votes
0 answers
30 views

I screwed-up model selection but ended-up with a very good model; am I ok?

In a recent experiment, I made an oversight: I divided my data into training and testing sets and conducted cross-validation for model selection and hyperparameter tuning after having applied Boruta (...
Alek Fröhlich's user avatar
1 vote
0 answers
12 views

Model choice based on test/train/validation split [duplicate]

My question is very simple, but no matter where I look it up, it seems that I get another answer. Take a simple classification task. Let's say I trained a kNN, LDA and logistic regression on it for ...
Marlon Brando's user avatar
2 votes
1 answer
33 views

What's the relationship between "bias-variance tradeoff" and "consistent model selection"?

I'm very confused about the relationship between "bias-variance tradeoff" and "consistent model selection". Based on my current interpretation, the ultimate goal of taking care of ...
ExcitedSnail's user avatar
  • 2,718
3 votes
1 answer
186 views

Reduce the model sequentially

I was given an ANOVA table and asked to reduce the model sequentially. I searched the online resources say: When reducing the model sequentially, you typically start by assessing the significance of ...
Matata's user avatar
  • 671
2 votes
0 answers
32 views

Confusion about Mallows' Cp

I am trying to use Mallows' $C_p$ to select linear regression models. I have been reading the excellent text by Cosma Shalizi at https://www.stat.cmu.edu/~cshalizi/TALR/TALR.pdf (page 323 to 327). ...
Frank De Geeter's user avatar
1 vote
1 answer
29 views

Selection of best VARX model using VAR() in R

I have 9 variables (all stationary) grouped into five different datasets (each set has 4 common variables and one different). How can I evaluate which is the best VARX model? I'm using ...
Alfonso's user avatar
  • 11
3 votes
0 answers
33 views

What critical level to use in diagnostic tests for model selection in forecasting?

I have been reading Hyndman & Athanasopoulos "Forecasting: Principles and Practice" (newest edition here) recently, and I noticed something that I regard as a possible inconsistency. On ...
Richard Hardy's user avatar
1 vote
1 answer
40 views

What modeling approach should i use AR, MA, ARMA?

those are my ACF and PACF plots for my time series after two differentiating. I watched a couple of tutorials but I cannot figure out what method I am supposed to use. Also, an interpretation of the ...
antekkalafior's user avatar
0 votes
0 answers
25 views

How to fit a dataset like this, and what's the recommended evaluate metrics for it

the dataset seems like non-linear, is there any recommended way to fit the datatset? since it's a non-linear regression problem, what's the correct way to evaluate the model's prediction? is the MSE ...
Wuuu's user avatar
  • 1
0 votes
0 answers
12 views

Online mixture inference; better alternatives than windowed EM?

I have an online Gaussian mixture estimation problem that I would appreciate some input on. To be more precise, I have a stream of scalar observations $x_1, x_2, \dotsc$ arriving over time which are ...
ummg's user avatar
  • 145
1 vote
1 answer
20 views

Is mutual exclusivity important for an A/B test for an audience selection method?

Say I want to measure whether a set of business rules is better than random at identifying customers most likely to respond to an email. The steps are: Take the entire population of 200 people and ...
djs's user avatar
  • 11
0 votes
0 answers
12 views

Comparing and selecting models, constructing objective function (complexity, prior knowledge on the distribution of parameter values)

We have a set of models that were derived using some fitting routine that optimizes parameter values utilizing $\chi^2$ for a given model. model1 has 100 parameters, model2 has 99 parameters, ... ...
twistfire's user avatar
  • 113
1 vote
1 answer
21 views

Interpreting AIC relative likelihoods ( qpcR::akaike.weights() )

I want to ensure that I am correctly interpreting AIC relative likelihood (RL) scores, specifically those returned by qpcR::akaike.weights$rel.LL. For example, I ...
PhelsumaFL's user avatar
7 votes
3 answers
1k views

Is parsimony crucial for statistical inference?

This question is based on using a regression for statistical inference (not prediction). I have conducted hierarchical (logistic mixed effects) regression. The first model includes the predictors of ...
SilvaC's user avatar
  • 512
1 vote
0 answers
22 views

Finding optimal combination of covariates using cross validation

I have a logistic mixed model (lme4 package in R). I want to assess whether participants scores on the measures 'sumspq', 'sumpdi', and 'sumcaps' significantly affect the difference in performance ...
SilvaC's user avatar
  • 512
4 votes
2 answers
149 views

Generalized additive model: Variable & model selection

I know this type of question has been asked many times before, so I apologize for re-posting about it. I bring it up again because it's been taught in one of my courses of study and I want to make ...
Nate's user avatar
  • 1,499
1 vote
0 answers
49 views

How many lags of independent variables to use

I have a panel dataset (26880 observations) of individual decisions ($y_{it}$, a categorical variable) which depends on signals ($x_{it}$). I am trying to find out how many past signals individuals ...
jasmine's user avatar
  • 367
4 votes
1 answer
55 views

How to choose the best one model from ARIMAX, ARCH/GARCH and VAR?

Now I have 3 models to find what economic factors have an effect on gold price: ARIMAX model ARCH/GARCH model VAR model What is the tool to find the best model? In linear regression, we can ...
Susan's user avatar
  • 41
0 votes
0 answers
17 views

F-statistic and p-values plots with lag

I am currently working on multivariate predictive models and have generated several diagrams to represent F-statistics and p-values for different models across various lags. I would greatly appreciate ...
Creative T's user avatar
7 votes
2 answers
121 views

How can I get a best model? An exploratory LMM

I'd like to inquire about the linear mixed model and its application to my dataset. The dataset comprises a dependent variable (DV) denoted as V, alongside three ...
TKw's user avatar
  • 71
11 votes
7 answers
2k views

Why do we use Linear Models when tree based models often work better than linear models?

In Supervised Machine Learning, and specifically on Kaggle, it is usually seen that tree models often outperform linear models. And even in the tree-based models, it is usually XGBoost that ...
letdatado's user avatar
  • 325
1 vote
1 answer
35 views

Auto arima capital D

I've got time series 3 years long, there is seasonal uplift during December - but it's not so clear. Seasonal test fails. I train model twice without setting any parameter: ...
voncuver's user avatar
3 votes
2 answers
72 views

Variable selection in logistic regression [duplicate]

So I'm trying to make a multivariate logistic regression model in R studio. I'm not sure how to go about this. What seemed to make sense to me was to model every predictor against the response ...
AdmiralMunson's user avatar
1 vote
1 answer
70 views

How can Null model likelihood be higher than Fitted model likelihood

As far as I know, when fitting a GLM, the fitted model should always have a higher likelihood compared to the null model (with only an intercept) for the same training set. When I run a small ...
Kozolovska's user avatar
  • 1,333
4 votes
0 answers
26 views

Why do model selection criteria (xICs, etc) not explicitly incorporate a loss function?

Model Selection and Multimodel Inference by Burnham and Anderson notes that TIC, AIC, AICc and QAICc are based on K-L distance between a given model and true model. Also BIC is in a sense based on ...
Mohan's user avatar
  • 865
0 votes
0 answers
45 views

What kind of series is this? Time series Model Choice

I am trying to model a series (mainly as self-education, as a way of deepening my time series knowledge). The series is percentage growth in GDP per capita for the United States, taken from the World ...
Lindsay's user avatar
0 votes
0 answers
23 views

What exactly is the right approach when trying to find OOS MSE when using linear lasso regression?

This isn't a question where I have a code example to provide. It is more of an informal question about what to do between 2 options. Assume I have some data and my goal is to fit a model using the ...
Donk's user avatar
  • 1
1 vote
0 answers
33 views

Hyperparameter selection after nested cross-validation and making comparisons with DeLong's test

I have already read all the associated questions on the topic but couldn't find a clear answer. I initially split my data into training (80%) and hold-out testing (20%). Then, I am performing nested ...
user22409235's user avatar
2 votes
1 answer
85 views

pdf vs probability vs likelihood [duplicate]

How to compute the log likelihood? Let's take a simple example using a normal distribution and scipy to do the work. Assuming X is the data, and the normal distribution as the model (...
brice rebsamen's user avatar
1 vote
1 answer
41 views

Calculate (quasi) AIC for mixed-effect baseline-category (multinomial logit) model

I am doing a discrete choice experiment where respondents are presented with different patient profiles, and for each profile, respondents need to choose one (out of three) treatment options. An ...
Trang Hien's user avatar
4 votes
1 answer
33 views

Does adding a random intercept for subject address confounding variables within subjects (e.g. sex or age)?

Let's say I am interested in identifying associations between a blood protein and disease activity, but I have multiple measurements per subject. Based on a literature review, I expect sex differences ...
HarD's user avatar
  • 207
1 vote
0 answers
45 views

Specifying parameters for SARIMAX model with significant ACF / PACF at tails

I have hourly data that has a period of 1 day or 24 hours / time steps and I hope to do short term forecasting for a few days in advance. The ACF of the raw time series was periodic (see last figure) ...
Yandle's user avatar
  • 1,149
1 vote
1 answer
40 views

LRT & AIC comparison, selecting a model fitted by weighted LS

I am a completely newbie in statistics and I want to ask a couple of questions about Akaike Information Criteria & Likellihood Ratio Test for particular application. I am trying to fit data using ...
twistfire's user avatar
  • 113
2 votes
1 answer
71 views

R-squared vs adjusted R-squared in Hierarchical multiple regression

In hierarchical multiple regression (not to be confused with hierarchical linear models that account for variance components), you add model terms by block. The fit of the new model is measured by the ...
Migs F's user avatar
  • 21
0 votes
0 answers
34 views

BMA formula with BIC

I am interested in using Bayesian modele averaging as a selection creteria (BMA) vs AIC. I read that BMA is widely implemented in clustering models. Suppose that we need to fit M models to a data and ...
Alice's user avatar
  • 640
0 votes
0 answers
16 views

Model Selection on Coxme

I am building a "time to event" model using the coxme package in R. I have a lot of mixed effects (15) that I want to input into a global model and find the best fit for - is there a model ...
michellemoyah's user avatar
0 votes
0 answers
26 views

Total generalized variance for Box-Cox transformed components

I have a couple Gaussian mixture models where each component comes from (component-wise) Box-Cox transformed data. These models do not describe the same data: the individual components are selected ...
ladislaw94's user avatar
1 vote
1 answer
200 views

Which regression model would you choose?

Which regression model would you choose to model the following flood damage data? The variables are x1=water height, x2=dike height and x3=flood damage. The following plot shows how the flood damages ...
Sjafnargata's user avatar
1 vote
1 answer
34 views

train / validation / test split problem

Suppose that I have created train/validation/test splits for model building. I optimized the hyperparameters using the validation set and chose the parameter values which gave the highest accuracy. To ...
Sanyo Mn's user avatar
  • 1,252
2 votes
1 answer
69 views

Nested models: should we use criteria like adjusted R squared, AIC/BIC, Mallows' Cp or the F-test?

I am confused about which one to use for, say testing $ H_0: \beta_1 = \beta_2 $. The F-test for comparison of residual sum of squares can be used here, as can things like adjusted $R^2$, AIC/BIC or ...
Ariel4778's user avatar
0 votes
0 answers
22 views

Is there any bias introduced by evaluating a model and decisions based on this model on the same data set?

As an example, let's say we have some financial time series such as closing prices of some stock and we would like to evaluate the ability of different models to forecast future closing prices as well ...
QMath's user avatar
  • 327
2 votes
1 answer
46 views

Model Building Process Without Feature Names/Domain Knowledge

I am working with a dataset of $\approx 10^7$ samples and $\approx 120$ features. All samples have a binary classification $0 - 1$. I am attempting to build a model that minimizes out-of-sample error ...
Bepop's user avatar
  • 307
0 votes
0 answers
17 views

Reporting policy effect for specific subpopulation - how to interract?

I am seeking guidance on model selection when reporting a causal policy effect for a specific subgroup of the panel. I am open to any form of help: paper, maths, intuitions... Below my setup, please ...
axel365's user avatar
  • 21
1 vote
0 answers
29 views

Best Strategy for Model Training & Selection (Spoiler: Should I Re-Train?)

After a discussion with some colleagues, I've realized we've different views on which is the go-to strategy for model training. Strategy A: Train-Validation-Test Split and Final Model Selection ...
rusiano's user avatar
  • 514
4 votes
0 answers
31 views

Can cross-validation be involved in model-building rather than validation?

I have a general idea in mind that would go like this: randomly split the data into training/testing build a model on the training data by choosing from among candidate predictors evaluate it on the ...
Dave's user avatar
  • 2,611
0 votes
0 answers
25 views

How do you use ordinal response data with several random effects to do robust hypothesis testing?

I want to explore the roles of lung presence and habitat on tadpoles' ability to tolerate low oxygen levels. My experimental design generated several measurements of "responsiveness" (on an ...
Jackson_P_tadpole_devotee's user avatar
0 votes
0 answers
65 views

Tiny partial effects but high R² in GAMs - and the other way around

I am currently running GAMs with mgcv package and am trying to find a good one by looking at summary() and the visual outputs ...
user_20201213's user avatar
1 vote
1 answer
16 views

Associating a constant value over time with predictor variables

Question: How do you model a static response variable (change between baseline and final timepoint) with a variable that changes over time? Background: A person's inflammation levels are measured at 0,...
HarD's user avatar
  • 207

1
2 3 4 5
40