Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

2
votes
0answers
32 views
+100

ARMA-GARCH model selection / fit evaluation

I'm trying to fit an ARMA-GARCH model to a data set of FTSE 100 log returns. However, I'm not able to find a well-fitting model. Below are the ACF and PACF of the log return series and the ACF of the ...
0
votes
0answers
15 views

AIC for multiple nonlinear regression models

How do we got about using AIC for multiple nonlinear regression models ? For example: If i have a dataset with N instances, and they can be explained by a collection of 3 models where each model has ...
3
votes
0answers
31 views

The use of the negative binomial dispersion parameter in model selection…?

I'm doing model selection, analysing the effect of a number of variables on the number of shoots browsed by deer, using the number of shoots available as an offset variable. My data distribution is ...
0
votes
0answers
20 views

How to read K-Fold Cross Validation results?

If I have two models to be validated, how I could figure which model is the best? Is it the one who has bigger score, or the smaller one? Any reference for in-depth explanation and example for k-fold ...
2
votes
0answers
20 views

Lag Selection in an unbalanced panel in R

How to determine appropriate number of lags in an unbalanced panel? Thanks.
0
votes
0answers
12 views

Computing weighted AIC scores [duplicate]

I am trying to compute the weighted AIC using the example posted here as a basis: $$ w_i = \frac{e^{(-0.5\mathsf{\Delta}_i)}}{\sum_{r=1}^Re^{(-0.5\mathsf{\Delta}_i)}}. $$ where ${\Delta}_i$ is the ...
1
vote
1answer
32 views

Proper use of model inference (AIC) (Burnham and Anderson) - when to explore more models

I am starting an analysis, for which I have a binomial response variable (species relative abundance) and continuous predictors (habitat variables). I have done some data exploration, and there is ...
0
votes
0answers
34 views

Use adjusted R-squared to select between regression models

I use the same sample to run two regressions. Both regressions have the same dependent and independent variables except in one regression the dependent variable and one of the independent variables ...
1
vote
2answers
103 views

Model for comparison of two subsets of the same data

I am looking to perform an analysis on a subset of the data and compare it to a larger subset. My data is primarily categorical and the dependent variable is binary. I want to compare $y^*= \beta ...
1
vote
1answer
28 views

Bayesian model selection, functional form of variance

I'm working on a linear regression model of the form, $$y = X\beta + \epsilon(X) $$ where each $$\epsilon_i \sim N(0, \sigma^2_i)$$ My variance term is depends on a subset of the regressors $X$. ...
3
votes
1answer
63 views

R-squared as criterion to choose between linear and non-linear regression

I am working in some regression models to forecast opinions based on general demographic characteristics, and I'm not sure how to choose between linear regression and curve estimation (I'm using SPSS ...
0
votes
1answer
87 views

Model selection and performance evaluation with different sample sizes

Suppose there are K experimental units. Each unit is associated with its own dataset consisting of 400 observations. For each unit, we set up a two-sample test, 200 vs 200. Because of a large sample ...
0
votes
1answer
28 views

Using LOOCV, AIC for Weighted Multiple Regression Model Selection?

I am currently attempting to determine the most predictive weighted multiple linear regression model to use and am trying to figure out the best combination of variables to use in the model. My first ...
0
votes
0answers
27 views

General to specific: t-stat, Akaike, Schwarz, and Adjusted R-squared

Specifying a linear model from general to specific i find that removing regressors corresponding to insignificant coefficients actually makes the adjusted r-squared, the Akaike and the Schwarz stats ...
1
vote
1answer
35 views

How to choose the order of a GARCH model?

In order to model time series with GARCH models in R, you first determine the AR order and the MA order using ACF and PACF plots. But then how do you determine the order of the actual GARCH model? ...
1
vote
0answers
32 views

Directed acyclic graphs in regression model

I am using DAGs to select best set of variables for my logistic regression analysis. Assessment of DAG includes one exposure, number of covariates and an outcome variable. I have not found any ...
1
vote
1answer
67 views

Backward selection (with fastbw) in penalized logistic regression

I have a dataset with more than 20 predictors and a single binary response variable. With only $n=181$ observations (64 deaths, 117 survivors), I decided to apply penalized logistic regression to ...
0
votes
0answers
14 views

A Binary Classification to Distinguish two Different Models?

I have two functions, a step function $f(x)$ and an inverse exponential function $g(x)$. Together, they explain virtually all the data when combined as a piecewise function. Some of the data points ...
1
vote
2answers
69 views

Is validation set always necessary?

Lets say I did the following steps: Used some separate development set to select some features. Decided a priori to use only one learning algorithm (SVM) with only default parameter values. Trained ...
0
votes
0answers
47 views

TIC criterion for normal regression model

I'm looking for the application of the TIC criterion in r. The TIC is an adaptation of the AIC criterion where the penalty term is replaced by the trace of the score and the Fisher information matrix ...
2
votes
0answers
71 views

R: Model selection with categorical variables using leaps and glmnet

I have a linear model containing a few continuous variables and four categorical variables, each represented by 12, 3, 4, and 5 dummy variables respectively. When using model selection criteria such ...
3
votes
2answers
80 views

Model Selection in Statistics

I have been told not to look at significance level, or not to use forward/backward selection using BIC/AIC for model selection. Let's say, I have 100 survey data with 11 variables and I want to see ...
0
votes
0answers
12 views

How to choose between two models on the basis of the normalised posterior distributions?

Suppose you are given two normalised posterior densities $\pi_1(\theta|y)$ and $\pi_2(\theta|y)$, based on the data $y$, and arising from model 1 and model 2, respectively. You are asked to find ...
1
vote
2answers
53 views

How to estimate variance of classifier on test set?

I have a binary classification task for which I want to compare two different classification methods as well as hyper-parameters for each. I have used k-fold cross-validation (k = 5) to obtain k ...
4
votes
1answer
39 views

BIC in Item Response Theory Models: Using log(N) vs log(N*I) as a weight

In IRT software packages and in the literature it is common to calculate the BIC as $$ \mathrm{BIC} = -2 \cdot \mathrm{logLik} + \log(N)\mathrm{Npars} $$ where $N$ is the number of rows in wide ...
2
votes
0answers
21 views

Data Assumptions for AIC model comparisons

I recently started digging into statistical information criteria, more specifically the Akaike Information Criterion. As the literature I have read so far does not cover this, I was wondering whether ...
0
votes
0answers
31 views

Interpretation of linear and quadratic interactions

I'm working on building and interpreting multilevel models. In one, I'm predicting binge episodes over time and using a linear time by baseline characteristic interaction as well as a quadratic time ...
2
votes
1answer
75 views

What is the best statistical model for my binary outcome variable?

My hypothesis is: As the experimental count variable increases, the probability that the binary dependent variable equals 1 increases. I expect both the independent and dependent variables to be not ...
2
votes
1answer
133 views

Lag selection for Augmented Dickey Fuller test

Apologies in advance, I am a beginner so these questions might be quite simple. I am testing log real exchange rates for unit root stationarity for EU15 countries. I was wondering what is the best way ...
0
votes
0answers
23 views

information criterion vs. log likelihood ratio

I am using Schwarz Bayesian Criterion (SBC) and Log likelihood ratio test for selecting the most appropriate model from a set of four time series regression models. The models are nested models. These ...
2
votes
1answer
52 views

Information criterion for selecting sample size when modeling tails

I want to model the left tail of an unknown distribution with a Generalized Pareto distribution. Somehow I have to select how much of the tail to model. I am wondering if it is possible to create an ...
2
votes
1answer
87 views

Constraints on the Coefficients of a Seasonal ARIMA Model (Possible Software Bug ITSM)

I am attempting to fit a seasonal ARIMA models using ITSM software. The following is the model. ARIMA$(1,1,0)\times(1,1,0)_{12}$: $\phi(B) \Phi(B^{12}) = (1-\phi B)(1-\Phi B^{12})=1-\Phi ...
1
vote
0answers
41 views

Regression with dependent variable which ranges from -1 to 1

I performed a series of Pearson correlations which give me as expected values between -1 and 1 (actually very few below zero). I'd like now to see if some factors are linked to these correlation ...
0
votes
0answers
46 views

Is this the wrong way to do cross-validation?

I am building an ARIMA model and did a grid search to find which values to use for my AR and MA components using the AIC criteria (this was using all of my data). The results are in this graphic: ...
6
votes
0answers
100 views

When should I be worried about the Jeffreys-Lindley paradox in Bayesian model choice?

I am considering a large (but finite) space of models of varying complexity which I explore using RJMCMC. The prior on the parameter vector for each model is fairly informative. In what cases (if ...
1
vote
1answer
19 views

Duplicate AICc values for multiple models with interactions

I am going through a model selection process with a mixed-model with 3 variables: A, B, and C. B and C are orthogonal factors. B or C may interact with A, so my full model would be: fixed: ...
2
votes
0answers
53 views

ARX model selection

I have an autoregressive model with exogenous variables: $S_{t} = \sum_{i=1}^{p} a_i S_{t-i} + \sum_q \sum_{i=1}^{r} b^q_i X^{q}_{t-i}$ where $S_t$ is the signal I want to predict and $X^q_t$ the ...
1
vote
0answers
3 views

Model comparison using different measures for the independent variables

I am doing a scale development study and would like to compare the newly developed measurement scale with an existing measure for the same construct and determine which scale better explains a set of ...
2
votes
0answers
29 views

The correct use of tensor product in gam (mgcv) function

I want to resurrect a question that I asked two months ago (Comparing gam models using ti( )), but adding more explanations. The aim of my analyses was to compare several gam models with different ...
3
votes
1answer
139 views

How to argue omitted variable problem is alleviated?

Is there any ways to argue that the omitted variable problem is alleviated after adding a new variable to the model? Right now I'm basically just saying that adding this new variable significantly ...
2
votes
1answer
74 views

Can you compare different functional models using Akaike criterion?

I have two regression models. One is a simple linear regression model ($y$ is regressed on $x_i$'s), while the other is a double log model (log of $y$ is regressed on log of the $x_i$'s). They have ...
1
vote
1answer
88 views

Fit data to a bivariate function

I want to fit my (x,y,z) data points to a function. You can see the data on Fig.1. The data is symmetric along the main diagonal. To understand my data I have studied (y,z) curves at different ...
1
vote
1answer
228 views

Model selection and comparison in GAMM using R (mgcv)

I'm fitting a GAMM with correlation structure, using a non-Gaussian family. Here's an example of my global model: ...
0
votes
0answers
10 views

Should information crtieria be applied to training or validation data?

Information criteria for selecting models seem to be applied to training data in general. Could they also be applied to validation data to decide the most predictive and simple model, or is this ...
0
votes
1answer
19 views

What are comments about the model specification when significance levels decrease in the multivariate model?

So I run the multivariate model and the significance levels decrease (i.e., larger p values) for almost all predictors. The extent to which they decrease vary, some may go from <0.001 to ...
0
votes
0answers
23 views

negative AIC or positive AIC? [duplicate]

I have calculated AIC by using R-studio to compare models. However, I got the following both negative and positive AIC AIC 8.52 0.41 -7.70 -5.84 -3.84 -2.10 Should I select the negative AIC or ...
1
vote
0answers
37 views

Help with R packages to use in my spatial discrete choice model

I am at a loss as to which R package and approach to use for my modeling. The background is: I have a transportation cross-sectional survey dataset combined with many other GIS datasets. I am ...
1
vote
0answers
16 views

Matching a query distribution to a family of template distributions

I was turning over a hypothetical question in my head: Suppose I have a set of template probability distributions, let's say each giving the probability of the occurrence of certain objects like ...
3
votes
1answer
82 views

Sample size when fitting categorical survey data

I have a model which fits data from repeated surveys: at time $t$, a number $n_t$ respondents is asked a question and can give one of $K$ answers ($k=1, ..., K$). This is repeated $T$ times ($t = 1, ...
1
vote
1answer
27 views

Model selection and assumed parameters in models

Suppose there are four models: Model 1: $y = ax$ Model 2: $y = ax^2$ Model 3: $y = a\sqrt{x}$ Model 4: $y = ax^\theta$ Model 4 is the most complex model with two parameters (the others have ...