# Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

1,876 questions
Filter by
Sorted by
Tagged with
1 vote
21 views

### How many knots for a spline fit of baseline hazard?

I am reading Tutz & Schmid "Modeling Discrete Time-to-Event Data" (2016) section 5.1 Smooth Baseline Hazard. On p. 108 they suggest two strategies for expanding time varying parameters ...
• 57.8k
1 vote
32 views

### Why model order selection is a big problem in statistics?

I’m learning statistical signal processing for my studies. I was doing a bit of literature review on model order selection and I didn’t want to miss out on techniques that I might not have seen. I ...
• 111
14 views

### How I can choose best regression model on my data? [closed]

[I want to find the best fit curve or model for my data and my data follows the following pattern. I have tried the curve fitting tool on Matlab but no model fits precisely on the data. Could you ...
1 vote
28 views

### Normalising likelihood for BIC/AIC calculation

I am running some model inference using AIC and BIC. My problem is that when I go and calculate the (maximum) loglikelihoods of my models, they are usually really high (range between 4700 and 1400 ...
• 13
18 views

### How to select optimal model based on nested cross-validation

I've been looking into nested cross validation and I asked questions here, but there is background information that I don't fully grasp. My situation is this: I have a dataset. I have three candidate ...
• 111
16 views

### Nested cross-validation: different hparams for each outer fold?

Suppose I have a dataset and two estimators, each with different hyperparameters. I want to select the best estimator and make sure it generalizes well. Therefore, I need to use nested cross-...
• 111
20 views

### What is the best model selection method for high-dimensional linear regression?

Model selection (best subset selection) in linear regression is quite important in many applications. Among the methods belonging to different frameworks such as information criterion, hypothesis ...
• 31
65 views

### R squared comparison

I have 5 features in my data. The R squared value when I use features 1,2, and 3 is $x$ and the R squared value when I use features 1,3, and 4 is $x + 0.1.$ Does this mean my second model is better ...
31 views

### Statistical tests vs model selection

I'm analyzing health data with case reports which contain patient information including diagnosis, age, sex, location etc. The sample sizes are not very large. I want to explore the data for ...
12 views

### Bayesian Model selection vs Model comparison

Is anyone aware of any articles or book chapters about the distinction between model selection and model comparison in bayesian multilevel modeling? I am fitting several competing growth models using ...
1 vote
30 views

### CV for Model selection - Why is significance testing not needed?

Say we have a model and some hyper-parameter with L values, and our goal is model selection. A k-fold CV outputs L accuracies (each accuracy is an average over K values). The best model corresponds to ...
• 995
16 views

### Model with lower AIC has violated assumptions of normality and homoscedasticity

I have a repeated measures dataset with multiple plants measured every month. My DV is growth, IVs are treatment and precipitation. Since I have measured by month, growth in my data has a high ...
16 views

### Using AIC to compare mediation models with inverted IV and mediator

We have collected correlational data, and are trying to figure out which model could best explain our data. We have two competing mediation models: DV explained by A, mediation by B DV explained by ...
20 views

### compare model fit logistic regression negative two times log likelihood

I'm trying to decide between two logistic regression models. I think I've used the negative two times log likelihood criterion before. My two models are not nested - can I still use that approach? ...
• 53
81 views

### How to reconcile that Bayesian model selection considers the entire parameter space of a model?

Background example: Coin Toss This is the standard example for Bayesian model selection (see, e.g., here). If you know this example, there is nothing new here: We want to find out whether a coin is ...
• 2,257
1 vote
17 views

### Best K-fold Segmentation Cross-validation

In image segmentation tasks, what validation metric is standard for model selection? I'm logging binary cross-entropy loss, Dice coefficient, and accuracy, but I don't know if I should use validation ...
• 111
1 vote
23 views

### What are training dataset, validation dataset and testing dataset in k-fold cross validation? [duplicate]

Question What are training dataset, validation dataset and testing dataset in K-fold cross validation? Which is used for model selection, validation dataset or testing dataset? Either validation ...
• 121
1 vote
90 views

### How to use priors on the parameter number with an information criterion (AIC, BIC, …)?

Example The example is made up because I hope that it’s more accessible than my actual problem. I want to determine the number of planets of a star. I have: data for some astronomical observable of ...
• 2,257
1 vote
13 views

### Temporal cross-validation in forecasting: model selection, hyperparameter tuning and comparison to independent forecast

I'm mainly working with time-series models and want to make sure to build the correct model selection process. Let's consider a forecasting problem and I have two model candiates, Model A and Model B. ...
• 11
753 views

### Is it required to train the model in entire data after cross validation?

I have a model trained as follows. ...
• 203
24 views

### How to apply the Jacobian correction to AIC for a transformed dependent variable when the transformation includes an independent variable?

I am comparing several OLS multivariate regression models of a dependent variable (we'll call it $Y$) using various transformations, some of which also involve one of the independent variables ($X_1$)....
24 views

### Having trouble finding the correct Auto ARIMA model to use

I have tried a lot of different models so far, but I'm having trouble finding the correct one. The problem is that RMSE is relatively high when I compare it to validation set. Here is the data: ...
• 167
32 views

### Backwards model selection, model average and model prediction. I'm lost

I am trying to see whether two grouping variables and their interaction (let's call them B and C, where B has two levels and C has three levels) affect the behavior of an animal (described by A, a ...
1 vote
112 views

### Volatility forecast using ARIMA GARCH

I am trying to forecast the volatility of the pair of currency USD/GBP. I am using python ans I used a GARCH model on the returns, but later on I found that I can fit an ARIMA-GARCH model to forecast ...
• 13
1 vote
26 views

### Comparing models with the betareg packpage

I'm having problems using betareg. I have a dataset that always shows different results depending on how I perform the analysis. I'm using ...
1 vote
71 views

### corrected AIC (AICc) assumes the model is univariate?

I'm considering using the AICc instead of the AIC to select models because my sample size is not much larger than my number of parameters (n=214, K=16 - which is not enough, according to Burnham and ...
268 views

### Valid model comparison/selection? Poisson, negative binomial, zero-inflated poisson, z-i negative binomial with AIC

I want to model counts of an event in a pre-post design. A sample dataset could look like this: ...
7 views

### How scale heterogeneity can be a source of correlation?

First off, I'd like to put this clearly that I am new to logit models and I am trying to learn their basics. So far, I have understood that in the sphere of multinomial logit models and MXL, there are ...
1 vote
16 views

### Does changing model due to overdispersion/underdispersion results in forking?

This is related to the post How much do we know about p-hacking "in the wild"?. The post does not clearly delineate the boundary between forking or not forking to me. Suppose I have a count ...
• 1,169
1 vote
101 views

### Model selection with AIC, what to do with the selected variables

If pvalues aren't useful to look at after performing AIC variable selection (Why are p-values misleading after performing a stepwise selection?), what should be the right thing to do in a scientific ...
71 views

### Model Selection vs. Ensemble Learning

Is model selection just a specific kind of ensemble learning, where ensemble learning is loosely defined as "combining multiple models in some capacity to hopefully get an improved model"? ...
2k views

### Widespread overfitting in health domain research?

I was reading about flaws with model selection techniques such as elimination based on significance and backwards selection via AIC (or similar) in the context of regression leading to inflated ...
• 335
1 vote
33 views

• 551
1 vote
100 views

### Model selection: Is AIC enough or should one compute the p-value in model selection (and if yes to how to do it?)?

I fitted 2 models with a python package (curve_fit function from scipy.optimize) one linear and one nonlinear. I want to compare those 2 models. I compared those to model by calculating the AIC using ...
• 551
1 vote
68 views

### Should I use cross validation for simple linear regression model?

I have a data set with 181 observations. I have 9 predictors and I have developed different regression models using ordinary linear regression and stepwise linear regression. Now I'm trying to decide ...
• 11
27 views

### Cross Validation on whole data for model comparision

I have good an imbalanced very small dataset (58 instances) and whould like to create a multiclass classification model. I'd like to use cross-validation in order to make the most out of the data I ...
• 109
596 views

### Model Selection: AIC/BIC and Cross-Validation gives different conclusion

In general, there vast number of ways to select model/feature in machine learning or statistics. For example, empirical method like Cross-Validation, Bootstrap methods or in sample penalty such as AIC,...
• 532
1 vote
23 views

### Selecting ARIMA based on ACF and PACF

I am facing this set of ACF and PACF graphs, of a log differentiated feature, which is proved stationary through unit root tests. However, I'm having trouble selecting the right ARIMA model: ARIMA(p,d,...
1 vote
34 views

### Interpret high p and q orders of GARCH models

I am currently working with GARCH models (sGARCH, E-GARCH and GJR-GARCH). My question is very general. I chose my p and q orders with the help of AIC criterion. The best models are sGARCH(2,3), E-...
• 11
40 views

### Model selection in simultaneous ARMA-GARCH modeling without AIC [closed]

How does one determine the mean model and the variance model in simultaneous ARMA-GARCH modeling without using AIC? Rather than two step look at ACF/PACF of residuals squared of ARMA to specify the ...
1 vote
153 views

### Getting the right seasonality PDQ for SARIMA model

I am a bit confused about how to identify the seasonal component of a SARIMA model. I am currently looking at forecasting rates (ocean carrier rates to be specific). The first thing I did was to ...
53 views

### Model selection criteria that represent a compromise between AIC and BIC

I am very familiar with the ideas and formula of the two popular model selection criteria AIC/AICc and BIC. When I use them for practical problems in chemometrics, the use of AIC/AICc often gives ...
• 33
55 views

### Is it possible that GEE and mixed effect GLM give contradicting answers? If so, which one should be trusted?

Is it possible that GEE and mixed effect GLM give contradictive answers in significance of covariates? I assume both GEE and GLM selects same covariates. If so, which one should be trusted? From ...
• 1,169