Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

Filter by
Sorted by
Tagged with
1 vote
1 answer
21 views

How many knots for a spline fit of baseline hazard?

I am reading Tutz & Schmid "Modeling Discrete Time-to-Event Data" (2016) section 5.1 Smooth Baseline Hazard. On p. 108 they suggest two strategies for expanding time varying parameters ...
1 vote
1 answer
32 views

Why model order selection is a big problem in statistics?

I’m learning statistical signal processing for my studies. I was doing a bit of literature review on model order selection and I didn’t want to miss out on techniques that I might not have seen. I ...
  • 111
0 votes
0 answers
14 views

How I can choose best regression model on my data? [closed]

[I want to find the best fit curve or model for my data and my data follows the following pattern. I have tried the curve fitting tool on Matlab but no model fits precisely on the data. Could you ...
  • 1
1 vote
2 answers
28 views

Normalising likelihood for BIC/AIC calculation

I am running some model inference using AIC and BIC. My problem is that when I go and calculate the (maximum) loglikelihoods of my models, they are usually really high (range between 4700 and 1400 ...
  • 13
0 votes
0 answers
18 views

How to select optimal model based on nested cross-validation

I've been looking into nested cross validation and I asked questions here, but there is background information that I don't fully grasp. My situation is this: I have a dataset. I have three candidate ...
  • 111
0 votes
0 answers
16 views

Nested cross-validation: different hparams for each outer fold?

Suppose I have a dataset and two estimators, each with different hyperparameters. I want to select the best estimator and make sure it generalizes well. Therefore, I need to use nested cross-...
  • 111
0 votes
0 answers
20 views

What is the best model selection method for high-dimensional linear regression?

Model selection (best subset selection) in linear regression is quite important in many applications. Among the methods belonging to different frameworks such as information criterion, hypothesis ...
2 votes
3 answers
65 views

R squared comparison

I have 5 features in my data. The R squared value when I use features 1,2, and 3 is $x$ and the R squared value when I use features 1,3, and 4 is $x + 0.1.$ Does this mean my second model is better ...
0 votes
0 answers
31 views

Statistical tests vs model selection

I'm analyzing health data with case reports which contain patient information including diagnosis, age, sex, location etc. The sample sizes are not very large. I want to explore the data for ...
  • 1
0 votes
0 answers
12 views

Bayesian Model selection vs Model comparison

Is anyone aware of any articles or book chapters about the distinction between model selection and model comparison in bayesian multilevel modeling? I am fitting several competing growth models using ...
1 vote
2 answers
30 views

CV for Model selection - Why is significance testing not needed?

Say we have a model and some hyper-parameter with L values, and our goal is model selection. A k-fold CV outputs L accuracies (each accuracy is an average over K values). The best model corresponds to ...
  • 995
0 votes
0 answers
16 views

Model with lower AIC has violated assumptions of normality and homoscedasticity

I have a repeated measures dataset with multiple plants measured every month. My DV is growth, IVs are treatment and precipitation. Since I have measured by month, growth in my data has a high ...
0 votes
0 answers
16 views

Using AIC to compare mediation models with inverted IV and mediator

We have collected correlational data, and are trying to figure out which model could best explain our data. We have two competing mediation models: DV explained by A, mediation by B DV explained by ...
  • 1
3 votes
1 answer
20 views

compare model fit logistic regression negative two times log likelihood

I'm trying to decide between two logistic regression models. I think I've used the negative two times log likelihood criterion before. My two models are not nested - can I still use that approach? ...
  • 53
2 votes
2 answers
81 views

How to reconcile that Bayesian model selection considers the entire parameter space of a model?

Background example: Coin Toss This is the standard example for Bayesian model selection (see, e.g., here). If you know this example, there is nothing new here: We want to find out whether a coin is ...
  • 2,257
1 vote
0 answers
17 views

Best K-fold Segmentation Cross-validation

In image segmentation tasks, what validation metric is standard for model selection? I'm logging binary cross-entropy loss, Dice coefficient, and accuracy, but I don't know if I should use validation ...
  • 111
1 vote
1 answer
23 views

What are training dataset, validation dataset and testing dataset in k-fold cross validation? [duplicate]

Question What are training dataset, validation dataset and testing dataset in K-fold cross validation? Which is used for model selection, validation dataset or testing dataset? Either validation ...
  • 121
1 vote
1 answer
90 views

How to use priors on the parameter number with an information criterion (AIC, BIC, …)?

Example The example is made up because I hope that it’s more accessible than my actual problem. I want to determine the number of planets of a star. I have: data for some astronomical observable of ...
  • 2,257
1 vote
0 answers
13 views

Temporal cross-validation in forecasting: model selection, hyperparameter tuning and comparison to independent forecast

I'm mainly working with time-series models and want to make sure to build the correct model selection process. Let's consider a forecasting problem and I have two model candiates, Model A and Model B. ...
8 votes
4 answers
753 views

Is it required to train the model in entire data after cross validation?

I have a model trained as follows. ...
  • 203
2 votes
0 answers
24 views

How to apply the Jacobian correction to AIC for a transformed dependent variable when the transformation includes an independent variable?

I am comparing several OLS multivariate regression models of a dependent variable (we'll call it $Y$) using various transformations, some of which also involve one of the independent variables ($X_1$)....
0 votes
0 answers
24 views

Having trouble finding the correct Auto ARIMA model to use

I have tried a lot of different models so far, but I'm having trouble finding the correct one. The problem is that RMSE is relatively high when I compare it to validation set. Here is the data: ...
0 votes
0 answers
32 views

Backwards model selection, model average and model prediction. I'm lost

I am trying to see whether two grouping variables and their interaction (let's call them B and C, where B has two levels and C has three levels) affect the behavior of an animal (described by A, a ...
1 vote
1 answer
112 views

Volatility forecast using ARIMA GARCH

I am trying to forecast the volatility of the pair of currency USD/GBP. I am using python ans I used a GARCH model on the returns, but later on I found that I can fit an ARIMA-GARCH model to forecast ...
  • 13
1 vote
1 answer
26 views

Comparing models with the betareg packpage

I'm having problems using betareg. I have a dataset that always shows different results depending on how I perform the analysis. I'm using ...
1 vote
1 answer
71 views

corrected AIC (AICc) assumes the model is univariate?

I'm considering using the AICc instead of the AIC to select models because my sample size is not much larger than my number of parameters (n=214, K=16 - which is not enough, according to Burnham and ...
4 votes
2 answers
268 views

Valid model comparison/selection? Poisson, negative binomial, zero-inflated poisson, z-i negative binomial with AIC

I want to model counts of an event in a pre-post design. A sample dataset could look like this: ...
0 votes
0 answers
7 views

How scale heterogeneity can be a source of correlation?

First off, I'd like to put this clearly that I am new to logit models and I am trying to learn their basics. So far, I have understood that in the sphere of multinomial logit models and MXL, there are ...
1 vote
0 answers
16 views

Does changing model due to overdispersion/underdispersion results in forking?

This is related to the post How much do we know about p-hacking "in the wild"?. The post does not clearly delineate the boundary between forking or not forking to me. Suppose I have a count ...
  • 1,169
1 vote
0 answers
101 views

Model selection with AIC, what to do with the selected variables

If pvalues aren't useful to look at after performing AIC variable selection (Why are p-values misleading after performing a stepwise selection?), what should be the right thing to do in a scientific ...
2 votes
1 answer
71 views

Model Selection vs. Ensemble Learning

Is model selection just a specific kind of ensemble learning, where ensemble learning is loosely defined as "combining multiple models in some capacity to hopefully get an improved model"? ...
19 votes
3 answers
2k views

Widespread overfitting in health domain research?

I was reading about flaws with model selection techniques such as elimination based on significance and backwards selection via AIC (or similar) in the context of regression leading to inflated ...
  • 335
1 vote
0 answers
33 views

Which kind of normalizations is better?

I want to do feature selection for a linear model. For this, each feature coloumn should be normalized, and a useful manner is substracting the mean and divided the l2 norm (to obtain unit norm): $\...
  • 11
0 votes
0 answers
7 views

Granular metrics for comparing model's biases

We are trying to compare two models performance. Mainly interested in understanding how these models impact our individual clients. For this we are generating a histogram for each model where the X ...
  • 289
0 votes
0 answers
19 views

Difference between Bayesian Information Criteria and Approximate Bayesian Computation as model selection

My question is not very technical and more like a discussion but I will be happy to have a technical input for the comparison b/w BIC and ABC. I am trying to understand and use the best model ...
3 votes
1 answer
80 views

Handling interactions when performing model selection for gam and result interpretation

I'm relatively new with gams, yet handling a complex dataset. I have a set of related questions and none of the posts on the topics fully answer my questions. Let's start with a simple question and ...
1 vote
0 answers
106 views

How to do deal with varying performance across a wide range of parameters?

We are working on a recommendation system. Our data comes from varying sources that make it natural to train a model for each source. Now we come to the crux. For each of the data sources we get ...
  • 93
3 votes
3 answers
301 views

Calculate AIC for both linear and non-linear models

I have data made of vectors $\textbf{x}$ and $\textbf{y}$. I want to predict $\textbf{y}$ with $\textbf{x}$ and a set of hyperparameters $a_{1, ..., 3}$ to be fitted with a linear and a nonlinear ...
  • 551
1 vote
1 answer
100 views

Model selection: Is AIC enough or should one compute the p-value in model selection (and if yes to how to do it?)?

I fitted 2 models with a python package (curve_fit function from scipy.optimize) one linear and one nonlinear. I want to compare those 2 models. I compared those to model by calculating the AIC using ...
  • 551
1 vote
0 answers
68 views

Should I use cross validation for simple linear regression model?

I have a data set with 181 observations. I have 9 predictors and I have developed different regression models using ordinary linear regression and stepwise linear regression. Now I'm trying to decide ...
  • 11
0 votes
1 answer
27 views

Cross Validation on whole data for model comparision

I have good an imbalanced very small dataset (58 instances) and whould like to create a multiclass classification model. I'd like to use cross-validation in order to make the most out of the data I ...
  • 109
6 votes
3 answers
596 views

Model Selection: AIC/BIC and Cross-Validation gives different conclusion

In general, there vast number of ways to select model/feature in machine learning or statistics. For example, empirical method like Cross-Validation, Bootstrap methods or in sample penalty such as AIC,...
  • 532
1 vote
0 answers
23 views

Selecting ARIMA based on ACF and PACF

I am facing this set of ACF and PACF graphs, of a log differentiated feature, which is proved stationary through unit root tests. However, I'm having trouble selecting the right ARIMA model: ARIMA(p,d,...
1 vote
0 answers
34 views

Interpret high p and q orders of GARCH models

I am currently working with GARCH models (sGARCH, E-GARCH and GJR-GARCH). My question is very general. I chose my p and q orders with the help of AIC criterion. The best models are sGARCH(2,3), E-...
0 votes
1 answer
40 views

Model selection in simultaneous ARMA-GARCH modeling without AIC [closed]

How does one determine the mean model and the variance model in simultaneous ARMA-GARCH modeling without using AIC? Rather than two step look at ACF/PACF of residuals squared of ARMA to specify the ...
1 vote
0 answers
153 views

Getting the right seasonality PDQ for SARIMA model

I am a bit confused about how to identify the seasonal component of a SARIMA model. I am currently looking at forecasting rates (ocean carrier rates to be specific). The first thing I did was to ...
3 votes
1 answer
53 views

Model selection criteria that represent a compromise between AIC and BIC

I am very familiar with the ideas and formula of the two popular model selection criteria AIC/AICc and BIC. When I use them for practical problems in chemometrics, the use of AIC/AICc often gives ...
  • 33
3 votes
1 answer
55 views

Is it possible that GEE and mixed effect GLM give contradicting answers? If so, which one should be trusted?

Is it possible that GEE and mixed effect GLM give contradictive answers in significance of covariates? I assume both GEE and GLM selects same covariates. If so, which one should be trusted? From ...
  • 1,169
0 votes
0 answers
114 views

The order of SMOTE, Feature selection, Model selection?

Please teach me if I am wrong. The appropriate order should be: SMOTE Feature selection (e.g., by using a wrapper method) Model selection (e.g., by selecting the model with highest AUC) Then ...
0 votes
1 answer
21 views

If a time-series achieves max-likelihood at GARCH(1,1), would EGARCH, or other GARCH variations achieve global maximum likelihood at p=1, q=1?

If I find that a time-series fits GARCH(1,1), would EGARCH, or other GARCH variations still be X-GARCH(1,1)?

1
2 3 4 5
38