Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

2
votes
1answer
15 views

Is it valid to rank goodness-of-fit based on the value of the Kolmogorov-Smirnov or Anderson-Darling test statistics?

When using Kolmogorov-Smirnov or Anderson-Darling goodness-of-fit tests, is it valid to claim that, because distribution X has a lower test statistic than distribution Y, distribution X is a better ...
20
votes
6answers
3k views

Should parsimony really still be the gold standard?

Just a thought: Parsimonious models have always been the default go-to in model selection, but to what degree is this approach outdated? I'm curious about how much our tendency toward parsimony is a ...
0
votes
1answer
22 views

Best candidate model using AIC or BIC equal to initial model used to generate simulated data?

For a given ARMA model (order and coefficients are known) we generate simulated data. Model is stationary and invertible. Then using this data, I want to find the best model by trying all combinations ...
2
votes
0answers
16 views

Can F-Test be used for arbitrary nested non-linear model selection

Suppose I have data consisting of pairs $(x_i,y_i)$. To this I want to fit a function $\hat{y}_i=f(x_i;a_1,\ldots,a_n)$, where $a_j$ are parameters. The functions nest in the sense that setting the ...
0
votes
1answer
31 views

Using Pearson's $R^2$ for model selection

I have a question about using $R^2$ as a "best fit" technique for cross-sectional (not time series) type data... Suppose you have a data set, and you're trying to fit a regression model to it. You ...
0
votes
0answers
20 views

Equivalent measure to Matthews correlation coefficient, MCC, for multiclass classification

Thanks in advance for the help. MCC gives a measure of the quality of a binary classifier. I'm looking for a similar measure that can be used for a multi-class classifier. Ultimately what I would ...
0
votes
1answer
13 views

develop minimum adequate model with correlated predictors

Could someone guide me what should my approach be regarding what predictors to include if they are correlated and how to develop my minimum adequate model. For e.g. lets say I have 10 predictors some ...
0
votes
0answers
28 views

Bayesian model selection in PyMC3

I am using PyMC3 to run Bayesian models on my data. I am new to Bayesian modeling but according to some blogs posts, Wikipedia and QA from this site, it seems to be a valid approach to use Bayes ...
0
votes
1answer
16 views

weighted glm model selection

Can AIC values between different weighed models be compared to select the best model (ie the model with the lowest weighted AIC)? For example, if my response variable is the 'Average Sales Per ...
2
votes
2answers
28 views

BIC selection yields much smaller model than AIC - can I use the likelihood ratio test to compare?

I'm trying to model the data (not make predictions) and am NOT using lasso for this, just want to know if my plan is somewhat reasonable here: I'm modelling for a "yes/no" response variable, so I ...
0
votes
0answers
12 views

Model selection of GEE using QIC: plausible models

I'm using GEE (generalized estimating equations) for the first time and selecting between multiple GEE using difference in QIC. The models differ in their independent variables. Here's my questions: ...
3
votes
3answers
85 views

Is it feasible to rank several unnested Generalized (Additive and Linear) Models by AIC score?

I have one response variable and a large number (>100) of explanatory variables. METHOD 1: I have completed one approach where the explanatory variables have been reduced via a PCA (in accordance ...
0
votes
0answers
39 views

ARIMA models for mortality modelling (Box-Jenkins methodology)

Fitting the Lee-Carter model of mortality to data provides a time series for the period-related effect, which is subsequently often modelled as an ARIMA(p,d,q) process in order to make forecasts. p,d ...
2
votes
1answer
106 views

ARMA-GARCH model selection / fit evaluation

I'm trying to fit an ARMA-GARCH model to a data set of FTSE 100 log returns (which I've uploaded here). However, I'm not able to find a well-fitting model. Below are the ACF and PACF of the log return ...
0
votes
0answers
19 views

AIC for multiple nonlinear regression models

How do we got about using AIC for multiple nonlinear regression models ? For example: If i have a dataset with N instances, and they can be explained by a collection of 3 models where each model has ...
3
votes
0answers
36 views

The use of the negative binomial dispersion parameter in model selection…?

I'm doing model selection, analysing the effect of a number of variables on the number of shoots browsed by deer, using the number of shoots available as an offset variable. My data distribution is ...
0
votes
0answers
27 views

How to read K-Fold Cross Validation results?

If I have two models to be validated, how I could figure which model is the best? Is it the one who has bigger score, or the smaller one? Any reference for in-depth explanation and example for k-fold ...
2
votes
0answers
22 views

Lag Selection in an unbalanced panel in R

How to determine appropriate number of lags in an unbalanced panel? Thanks.
0
votes
0answers
12 views

Computing weighted AIC scores [duplicate]

I am trying to compute the weighted AIC using the example posted here as a basis: $$ w_i = \frac{e^{(-0.5\mathsf{\Delta}_i)}}{\sum_{r=1}^Re^{(-0.5\mathsf{\Delta}_i)}}. $$ where ${\Delta}_i$ is the ...
1
vote
1answer
40 views

Proper use of model inference (AIC) (Burnham and Anderson) - when to explore more models

I am starting an analysis, for which I have a binomial response variable (species relative abundance) and continuous predictors (habitat variables). I have done some data exploration, and there is ...
0
votes
0answers
44 views

Use adjusted R-squared to select between regression models

I use the same sample to run two regressions. Both regressions have the same dependent and independent variables except in one regression the dependent variable and one of the independent variables ...
1
vote
2answers
110 views

Model for comparison of two subsets of the same data

I am looking to perform an analysis on a subset of the data and compare it to a larger subset. My data is primarily categorical and the dependent variable is binary. I want to compare $y^*= \beta ...
1
vote
1answer
29 views

Bayesian model selection, functional form of variance

I'm working on a linear regression model of the form, $$y = X\beta + \epsilon(X) $$ where each $$\epsilon_i \sim N(0, \sigma^2_i)$$ My variance term is depends on a subset of the regressors $X$. ...
3
votes
1answer
82 views

R-squared as criterion to choose between linear and non-linear regression

I am working in some regression models to forecast opinions based on general demographic characteristics, and I'm not sure how to choose between linear regression and curve estimation (I'm using SPSS ...
0
votes
1answer
98 views

Model selection and performance evaluation with different sample sizes

Suppose there are K experimental units. Each unit is associated with its own dataset consisting of 400 observations. For each unit, we set up a two-sample test, 200 vs 200. Because of a large sample ...
0
votes
1answer
36 views

Using LOOCV, AIC for Weighted Multiple Regression Model Selection?

I am currently attempting to determine the most predictive weighted multiple linear regression model to use and am trying to figure out the best combination of variables to use in the model. My first ...
0
votes
0answers
29 views

General to specific: t-stat, Akaike, Schwarz, and Adjusted R-squared

Specifying a linear model from general to specific i find that removing regressors corresponding to insignificant coefficients actually makes the adjusted r-squared, the Akaike and the Schwarz stats ...
1
vote
1answer
54 views

How to choose the order of a GARCH model?

In order to model time series with GARCH models in R, you first determine the AR order and the MA order using ACF and PACF plots. But then how do you determine the order of the actual GARCH model? ...
1
vote
0answers
47 views

Directed acyclic graphs in regression model

I am using DAGs to select best set of variables for my logistic regression analysis. Assessment of DAG includes one exposure, number of covariates and an outcome variable. I have not found any ...
1
vote
1answer
87 views

Backward selection (with fastbw) in penalized logistic regression

I have a dataset with more than 20 predictors and a single binary response variable. With only $n=181$ observations (64 deaths, 117 survivors), I decided to apply penalized logistic regression to ...
0
votes
0answers
14 views

A Binary Classification to Distinguish two Different Models?

I have two functions, a step function $f(x)$ and an inverse exponential function $g(x)$. Together, they explain virtually all the data when combined as a piecewise function. Some of the data points ...
1
vote
2answers
81 views

Is validation set always necessary?

Lets say I did the following steps: Used some separate development set to select some features. Decided a priori to use only one learning algorithm (SVM) with only default parameter values. Trained ...
0
votes
0answers
49 views

TIC criterion for normal regression model

I'm looking for the application of the TIC criterion in r. The TIC is an adaptation of the AIC criterion where the penalty term is replaced by the trace of the score and the Fisher information matrix ...
2
votes
0answers
87 views

R: Model selection with categorical variables using leaps and glmnet

I have a linear model containing a few continuous variables and four categorical variables, each represented by 12, 3, 4, and 5 dummy variables respectively. When using model selection criteria such ...
3
votes
2answers
89 views

Model Selection in Statistics

I have been told not to look at significance level, or not to use forward/backward selection using BIC/AIC for model selection. Let's say, I have 100 survey data with 11 variables and I want to see ...
0
votes
0answers
13 views

How to choose between two models on the basis of the normalised posterior distributions?

Suppose you are given two normalised posterior densities $\pi_1(\theta|y)$ and $\pi_2(\theta|y)$, based on the data $y$, and arising from model 1 and model 2, respectively. You are asked to find ...
1
vote
2answers
56 views

How to estimate variance of classifier on test set?

I have a binary classification task for which I want to compare two different classification methods as well as hyper-parameters for each. I have used k-fold cross-validation (k = 5) to obtain k ...
4
votes
1answer
42 views

BIC in Item Response Theory Models: Using log(N) vs log(N*I) as a weight

In IRT software packages and in the literature it is common to calculate the BIC as $$ \mathrm{BIC} = -2 \cdot \mathrm{logLik} + \log(N)\mathrm{Npars} $$ where $N$ is the number of rows in wide ...
2
votes
0answers
22 views

Data Assumptions for AIC model comparisons

I recently started digging into statistical information criteria, more specifically the Akaike Information Criterion. As the literature I have read so far does not cover this, I was wondering whether ...
0
votes
0answers
34 views

Interpretation of linear and quadratic interactions

I'm working on building and interpreting multilevel models. In one, I'm predicting binge episodes over time and using a linear time by baseline characteristic interaction as well as a quadratic time ...
2
votes
1answer
100 views

What is the best statistical model for my binary outcome variable?

My hypothesis is: As the experimental count variable increases, the probability that the binary dependent variable equals 1 increases. I expect both the independent and dependent variables to be not ...
2
votes
1answer
217 views

Lag selection for Augmented Dickey Fuller test

Apologies in advance, I am a beginner so these questions might be quite simple. I am testing log real exchange rates for unit root stationarity for EU15 countries. I was wondering what is the best way ...
0
votes
0answers
29 views

information criterion vs. log likelihood ratio

I am using Schwarz Bayesian Criterion (SBC) and Log likelihood ratio test for selecting the most appropriate model from a set of four time series regression models. The models are nested models. These ...
2
votes
1answer
54 views

Information criterion for selecting sample size when modeling tails

I want to model the left tail of an unknown distribution with a Generalized Pareto distribution. Somehow I have to select how much of the tail to model. I am wondering if it is possible to create an ...
2
votes
1answer
92 views

Constraints on the Coefficients of a Seasonal ARIMA Model (Possible Software Bug ITSM)

I am attempting to fit a seasonal ARIMA models using ITSM software. The following is the model. ARIMA$(1,1,0)\times(1,1,0)_{12}$: $\phi(B) \Phi(B^{12}) = (1-\phi B)(1-\Phi B^{12})=1-\Phi ...
1
vote
0answers
44 views

Regression with dependent variable which ranges from -1 to 1

I performed a series of Pearson correlations which give me as expected values between -1 and 1 (actually very few below zero). I'd like now to see if some factors are linked to these correlation ...
0
votes
0answers
53 views

Is this the wrong way to do cross-validation?

I am building an ARIMA model and did a grid search to find which values to use for my AR and MA components using the AIC criteria (this was using all of my data). The results are in this graphic: ...
6
votes
0answers
103 views

When should I be worried about the Jeffreys-Lindley paradox in Bayesian model choice?

I am considering a large (but finite) space of models of varying complexity which I explore using RJMCMC. The prior on the parameter vector for each model is fairly informative. In what cases (if ...
1
vote
1answer
22 views

Duplicate AICc values for multiple models with interactions

I am going through a model selection process with a mixed-model with 3 variables: A, B, and C. B and C are orthogonal factors. B or C may interact with A, so my full model would be: fixed: ...
2
votes
0answers
55 views

ARX model selection

I have an autoregressive model with exogenous variables: $S_{t} = \sum_{i=1}^{p} a_i S_{t-i} + \sum_q \sum_{i=1}^{r} b^q_i X^{q}_{t-i}$ where $S_t$ is the signal I want to predict and $X^q_t$ the ...