Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

0
votes
0answers
15 views

how to use the likelihood ratio test for model selection in the study with several subjects

In my study, I have 30 subjects, for each subject, I use likelihood ratio test to compare two models (nested logistic regression), and I get a chi-squared value and a p value like the result shown ...
0
votes
0answers
15 views

How to analyse the influence of 100 categorical or continuous predictors on one continuous response?

I am analysing a genetic dataset that consists of 288 individuals, 100 genetic markers as predictors and one continuos variable (day of death) as outcome. Each predictor has three categories or ...
1
vote
0answers
18 views

AICc penalising model complexity too highly?

My analysis uses a negative binomial GLMM with total revisits as the dependent variable, treatment (factor with 4 levels: 0ppb, 4.8ppb, 20ppb, 133ppb) and size as fixed effects and colony as a random ...
2
votes
1answer
43 views

Regression coefficients tests before or after model selection

I have a set of data containing 4 predictors (environmental conditions and animal size) and one predicted variable (animal growth rate). I want to fit a regression model to this data. I have two ...
0
votes
1answer
10 views

Correlation between standardized residuals and fitted values in a linear mixed effect model: Course of action?

I am fitting a linear mixed effect model in R with lme from nlmer, using the approach described in Zuur et al. "Mixed Effects ...
0
votes
1answer
41 views

Time series forecasting with R

I try to forecast my web visitors on the web site for 10 future days using time series. My time series is daily. I have used an auto.arima() model. Considering ...
3
votes
0answers
28 views

Leaving insignificant variables in a regression - Sources?

I'm aware that this question has been answered many times on this forum, however, I've not seen it accompanied by any sources. I'm looking for sources which support the inclusion of control variables ...
2
votes
2answers
28 views

Reasonable approach for modelling churn (survival) and choice of intervention campaign (multinomial regression)?

I've only recently moved into customer analytics, and would love to get some advice around designing a reasonable approach to modelling my data. I want to be able to predict customer churn (that is, ...
0
votes
0answers
45 views

Which model would you choose?

I am struggling with the model selection (ICs are very different and many insignificant models). Which model would you choose, and can you give a reason why you would choose this model? And what would ...
2
votes
1answer
83 views

What's the current thinking on selecting model complexity in the statistical community?

I was watching a recent presentation by a neural networks researcher who recommended using a model more complex than would be suggested by the data, and regularizing the life out of it. He said this ...
0
votes
1answer
35 views

Model selection in mixed-effects model with collinearity trouble

In a model aimed to assess the influence of land use measures on ecosystem functioning, I have one log-transformed dependent variable (the ecosystem function), and 5 fixed-effects independent ...
2
votes
2answers
37 views

Is it valid to rank goodness-of-fit based on the value of the Kolmogorov-Smirnov or Anderson-Darling test statistics?

When using Kolmogorov-Smirnov or Anderson-Darling goodness-of-fit tests, is it valid to claim that, because distribution X has a lower test statistic than distribution Y, distribution X is a better ...
20
votes
6answers
3k views

Should parsimony really still be the gold standard?

Just a thought: Parsimonious models have always been the default go-to in model selection, but to what degree is this approach outdated? I'm curious about how much our tendency toward parsimony is a ...
0
votes
1answer
27 views

Best candidate model using AIC or BIC equal to initial model used to generate simulated data?

For a given ARMA model (order and coefficients are known) we generate simulated data. Model is stationary and invertible. Then using this data, I want to find the best model by trying all combinations ...
2
votes
0answers
18 views

Can F-Test be used for arbitrary nested non-linear model selection

Suppose I have data consisting of pairs $(x_i,y_i)$. To this I want to fit a function $\hat{y}_i=f(x_i;a_1,\ldots,a_n)$, where $a_j$ are parameters. The functions nest in the sense that setting the ...
0
votes
1answer
33 views

Using Pearson's $R^2$ for model selection

I have a question about using $R^2$ as a "best fit" technique for cross-sectional (not time series) type data... Suppose you have a data set, and you're trying to fit a regression model to it. You ...
0
votes
0answers
26 views

Equivalent measure to Matthews correlation coefficient, MCC, for multiclass classification

Thanks in advance for the help. MCC gives a measure of the quality of a binary classifier. I'm looking for a similar measure that can be used for a multi-class classifier. Ultimately what I would ...
0
votes
1answer
16 views

develop minimum adequate model with correlated predictors

Could someone guide me what should my approach be regarding what predictors to include if they are correlated and how to develop my minimum adequate model. For e.g. lets say I have 10 predictors some ...
2
votes
1answer
78 views

Bayesian model selection in PyMC3

I am using PyMC3 to run Bayesian models on my data. I am new to Bayesian modeling but according to some blogs posts, Wikipedia and QA from this site, it seems to be a valid approach to use Bayes ...
0
votes
1answer
19 views

weighted glm model selection

Can AIC values between different weighed models be compared to select the best model (ie the model with the lowest weighted AIC)? For example, if my response variable is the 'Average Sales Per ...
2
votes
2answers
41 views

BIC selection yields much smaller model than AIC - can I use the likelihood ratio test to compare?

I'm trying to model the data (not make predictions) and am NOT using lasso for this, just want to know if my plan is somewhat reasonable here: I'm modelling for a "yes/no" response variable, so I ...
0
votes
0answers
28 views

Model selection of GEE using QIC: plausible models

I'm using GEE (generalized estimating equations) for the first time and selecting between multiple GEE using difference in QIC. The models differ in their independent variables. Here's my questions: ...
3
votes
3answers
88 views

Is it feasible to rank several unnested Generalized (Additive and Linear) Models by AIC score?

I have one response variable and a large number (>100) of explanatory variables. METHOD 1: I have completed one approach where the explanatory variables have been reduced via a PCA (in accordance ...
0
votes
0answers
44 views

ARIMA models for mortality modelling (Box-Jenkins methodology)

Fitting the Lee-Carter model of mortality to data provides a time series for the period-related effect, which is subsequently often modelled as an ARIMA(p,d,q) process in order to make forecasts. p,d ...
2
votes
1answer
133 views

ARMA-GARCH model selection / fit evaluation

I'm trying to fit an ARMA-GARCH model to a data set of FTSE 100 log returns (which I've uploaded here). However, I'm not able to find a well-fitting model. Below are the ACF and PACF of the log return ...
0
votes
0answers
23 views

AIC for multiple nonlinear regression models

How do we got about using AIC for multiple nonlinear regression models ? For example: If i have a dataset with N instances, and they can be explained by a collection of 3 models where each model has ...
3
votes
0answers
39 views

The use of the negative binomial dispersion parameter in model selection…?

I'm doing model selection, analysing the effect of a number of variables on the number of shoots browsed by deer, using the number of shoots available as an offset variable. My data distribution is ...
0
votes
0answers
28 views

How to read K-Fold Cross Validation results?

If I have two models to be validated, how I could figure which model is the best? Is it the one who has bigger score, or the smaller one? Any reference for in-depth explanation and example for k-fold ...
2
votes
0answers
25 views

Lag Selection in an unbalanced panel in R

How to determine appropriate number of lags in an unbalanced panel? Thanks.
0
votes
0answers
12 views

Computing weighted AIC scores [duplicate]

I am trying to compute the weighted AIC using the example posted here as a basis: $$ w_i = \frac{e^{(-0.5\mathsf{\Delta}_i)}}{\sum_{r=1}^Re^{(-0.5\mathsf{\Delta}_i)}}. $$ where ${\Delta}_i$ is the ...
1
vote
1answer
54 views

Proper use of model inference (AIC) (Burnham and Anderson) - when to explore more models

I am starting an analysis, for which I have a binomial response variable (species relative abundance) and continuous predictors (habitat variables). I have done some data exploration, and there is ...
0
votes
0answers
49 views

Use adjusted R-squared to select between regression models

I use the same sample to run two regressions. Both regressions have the same dependent and independent variables except in one regression the dependent variable and one of the independent variables ...
1
vote
2answers
120 views

Model for comparison of two subsets of the same data

I am looking to perform an analysis on a subset of the data and compare it to a larger subset. My data is primarily categorical and the dependent variable is binary. I want to compare $y^*= \beta ...
1
vote
1answer
33 views

Bayesian model selection, functional form of variance

I'm working on a linear regression model of the form, $$y = X\beta + \epsilon(X) $$ where each $$\epsilon_i \sim N(0, \sigma^2_i)$$ My variance term is depends on a subset of the regressors $X$. ...
3
votes
1answer
104 views

R-squared as criterion to choose between linear and non-linear regression

I am working in some regression models to forecast opinions based on general demographic characteristics, and I'm not sure how to choose between linear regression and curve estimation (I'm using SPSS ...
0
votes
1answer
107 views

Model selection and performance evaluation with different sample sizes

Suppose there are K experimental units. Each unit is associated with its own dataset consisting of 400 observations. For each unit, we set up a two-sample test, 200 vs 200. Because of a large sample ...
0
votes
1answer
41 views

Using LOOCV, AIC for Weighted Multiple Regression Model Selection?

I am currently attempting to determine the most predictive weighted multiple linear regression model to use and am trying to figure out the best combination of variables to use in the model. My first ...
0
votes
0answers
30 views

General to specific: t-stat, Akaike, Schwarz, and Adjusted R-squared

Specifying a linear model from general to specific i find that removing regressors corresponding to insignificant coefficients actually makes the adjusted r-squared, the Akaike and the Schwarz stats ...
1
vote
1answer
69 views

How to choose the order of a GARCH model?

In order to model time series with GARCH models in R, you first determine the AR order and the MA order using ACF and PACF plots. But then how do you determine the order of the actual GARCH model? ...
1
vote
0answers
55 views

Directed acyclic graphs in regression model

I am using DAGs to select best set of variables for my logistic regression analysis. Assessment of DAG includes one exposure, number of covariates and an outcome variable. I have not found any ...
1
vote
1answer
102 views

Backward selection (with fastbw) in penalized logistic regression

I have a dataset with more than 20 predictors and a single binary response variable. With only $n=181$ observations (64 deaths, 117 survivors), I decided to apply penalized logistic regression to ...
0
votes
0answers
14 views

A Binary Classification to Distinguish two Different Models?

I have two functions, a step function $f(x)$ and an inverse exponential function $g(x)$. Together, they explain virtually all the data when combined as a piecewise function. Some of the data points ...
1
vote
2answers
95 views

Is validation set always necessary?

Lets say I did the following steps: Used some separate development set to select some features. Decided a priori to use only one learning algorithm (SVM) with only default parameter values. Trained ...
0
votes
0answers
58 views

TIC criterion for normal regression model

I'm looking for the application of the TIC criterion in r. The TIC is an adaptation of the AIC criterion where the penalty term is replaced by the trace of the score and the Fisher information matrix ...
2
votes
0answers
100 views

R: Model selection with categorical variables using leaps and glmnet

I have a linear model containing a few continuous variables and four categorical variables, each represented by 12, 3, 4, and 5 dummy variables respectively. When using model selection criteria such ...
3
votes
2answers
94 views

Model Selection in Statistics

I have been told not to look at significance level, or not to use forward/backward selection using BIC/AIC for model selection. Let's say, I have 100 survey data with 11 variables and I want to see ...
0
votes
0answers
13 views

How to choose between two models on the basis of the normalised posterior distributions?

Suppose you are given two normalised posterior densities $\pi_1(\theta|y)$ and $\pi_2(\theta|y)$, based on the data $y$, and arising from model 1 and model 2, respectively. You are asked to find ...
1
vote
2answers
57 views

How to estimate variance of classifier on test set?

I have a binary classification task for which I want to compare two different classification methods as well as hyper-parameters for each. I have used k-fold cross-validation (k = 5) to obtain k ...
4
votes
1answer
42 views

BIC in Item Response Theory Models: Using log(N) vs log(N*I) as a weight

In IRT software packages and in the literature it is common to calculate the BIC as $$ \mathrm{BIC} = -2 \cdot \mathrm{logLik} + \log(N)\mathrm{Npars} $$ where $N$ is the number of rows in wide ...
2
votes
0answers
22 views

Data Assumptions for AIC model comparisons

I recently started digging into statistical information criteria, more specifically the Akaike Information Criterion. As the literature I have read so far does not cover this, I was wondering whether ...