Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

1
vote
0answers
15 views

LOOCV vs. AIC for Weighted Multiple Regression Model Selection

I am currently trying to determine the most predictive weighted multiple linear regression model to use. I don't have much formal statistical training, so I would greatly appreciate any help with the ...
0
votes
0answers
19 views

TIC criterion for normal regression model

I'm looking for the application of the TIC criterion in r. The TIC is an adaptation of the AIC criterion where the penalty term is replaced by the trace of the score and the Fisher information matrix ...
2
votes
0answers
36 views

R: Model selection with categorical variables using leaps and glmnet

I have a linear model containing a few continuous variables and four categorical variables, each represented by 12, 3, 4, and 5 dummy variables respectively. When using model selection criteria such ...
3
votes
2answers
70 views

Model Selection in Statistics

I have been told not to look at significance level, or not to use forward/backward selection using BIC/AIC for model selection. Let's say, I have 100 survey data with 11 variables and I want to see ...
0
votes
0answers
10 views

How to choose between two models on the basis of the normalised posterior distributions?

Suppose you are given two normalised posterior densities $\pi_1(\theta|y)$ and $\pi_2(\theta|y)$, based on the data $y$, and arising from model 1 and model 2, respectively. You are asked to find ...
1
vote
2answers
49 views

How to estimate variance of classifier on test set?

I have a binary classification task for which I want to compare two different classification methods as well as hyper-parameters for each. I have used k-fold cross-validation (k = 5) to obtain k ...
4
votes
1answer
31 views

BIC in Item Response Theory Models: Using log(N) vs log(N*I) as a weight

In IRT software packages and in the literature it is common to calculate the BIC as $$ \mathrm{BIC} = -2 \cdot \mathrm{logLik} + \log(N)\mathrm{Npars} $$ where $N$ is the number of rows in wide ...
2
votes
0answers
19 views

Data Assumptions for AIC model comparisons

I recently started digging into statistical information criteria, more specifically the Akaike Information Criterion. As the literature I have read so far does not cover this, I was wondering whether ...
0
votes
0answers
20 views

Interpretation of linear and quadratic interactions

I'm working on building and interpreting multilevel models. In one, I'm predicting binge episodes over time and using a linear time by baseline characteristic interaction as well as a quadratic time ...
2
votes
1answer
57 views

What is the best statistical model for my binary outcome variable?

My hypothesis is: As the experimental count variable increases, the probability that the binary dependent variable equals 1 increases. I expect both the independent and dependent variables to be not ...
2
votes
1answer
53 views

Lag selection for Augmented Dickey Fuller test

Apologies in advance, I am a beginner so these questions might be quite simple. I am testing log real exchange rates for unit root stationarity for EU15 countries. I was wondering what is the best way ...
0
votes
0answers
16 views

information criterion vs. log likelihood ratio

I am using Schwarz Bayesian Criterion (SBC) and Log likelihood ratio test for selecting the most appropriate model from a set of four time series regression models. The models are nested models. These ...
2
votes
1answer
48 views

Information criterion for selecting sample size when modeling tails

I want to model the left tail of an unknown distribution with a Generalized Pareto distribution. Somehow I have to select how much of the tail to model. I am wondering if it is possible to create an ...
2
votes
1answer
71 views

Constraints on the Coefficients of a Seasonal ARIMA Model (Possible Software Bug ITSM)

I am attempting to fit a seasonal ARIMA models using ITSM software. The following is the model. ARIMA$(1,1,0)\times(1,1,0)_{12}$: $\phi(B) \Phi(B^{12}) = (1-\phi B)(1-\Phi B^{12})=1-\Phi ...
1
vote
0answers
37 views

Regression with dependent variable which ranges from -1 to 1

I performed a series of Pearson correlations which give me as expected values between -1 and 1 (actually very few below zero). I'd like now to see if some factors are linked to these correlation ...
0
votes
0answers
42 views

Is this the wrong way to do cross-validation?

I am building an ARIMA model and did a grid search to find which values to use for my AR and MA components using the AIC criteria (this was using all of my data). The results are in this graphic: ...
6
votes
0answers
90 views

When should I be worried about the Jeffreys-Lindley paradox in Bayesian model choice?

I am considering a large (but finite) space of models of varying complexity which I explore using RJMCMC. The prior on the parameter vector for each model is fairly informative. In what cases (if ...
1
vote
1answer
17 views

Duplicate AICc values for multiple models with interactions

I am going through a model selection process with a mixed-model with 3 variables: A, B, and C. B and C are orthogonal factors. B or C may interact with A, so my full model would be: fixed: ...
2
votes
0answers
41 views

ARX model selection

I have an autoregressive model with exogenous variables: $S_{t} = \sum_{i=1}^{p} a_i S_{t-i} + \sum_q \sum_{i=1}^{r} b^q_i X^{q}_{t-i}$ where $S_t$ is the signal I want to predict and $X^q_t$ the ...
1
vote
0answers
3 views

Model comparison using different measures for the independent variables

I am doing a scale development study and would like to compare the newly developed measurement scale with an existing measure for the same construct and determine which scale better explains a set of ...
2
votes
0answers
19 views

The correct use of tensor product in gam (mgcv) function

I want to resurrect a question that I asked two months ago (Comparing gam models using ti( )), but adding more explanations. The aim of my analyses was to compare several gam models with different ...
3
votes
1answer
131 views

How to argue omitted variable problem is alleviated?

Is there any ways to argue that the omitted variable problem is alleviated after adding a new variable to the model? Right now I'm basically just saying that adding this new variable significantly ...
2
votes
1answer
66 views

Can you compare different functional models using Akaike criterion?

I have two regression models. One is a simple linear regression model ($y$ is regressed on $x_i$'s), while the other is a double log model (log of $y$ is regressed on log of the $x_i$'s). They have ...
0
votes
1answer
80 views

Fit data to a bivariate function

I want to fit my (x,y,z) data points to a function. You can see the data on Fig.1. The data is symmetric along the main diagonal. To understand my data I have studied (y,z) curves at different ...
1
vote
1answer
120 views

Model selection and comparison in GAMM using R (mgcv)

I'm fitting a GAMM with correlation structure, using a non-Gaussian family. Here's an example of my global model: ...
0
votes
0answers
8 views

Should information crtieria be applied to training or validation data?

Information criteria for selecting models seem to be applied to training data in general. Could they also be applied to validation data to decide the most predictive and simple model, or is this ...
0
votes
1answer
17 views

What are comments about the model specification when significance levels decrease in the multivariate model?

So I run the multivariate model and the significance levels decrease (i.e., larger p values) for almost all predictors. The extent to which they decrease vary, some may go from <0.001 to ...
0
votes
0answers
22 views

negative AIC or positive AIC? [duplicate]

I have calculated AIC by using R-studio to compare models. However, I got the following both negative and positive AIC AIC 8.52 0.41 -7.70 -5.84 -3.84 -2.10 Should I select the negative AIC or ...
1
vote
0answers
24 views

Help with R packages to use in my spatial discrete choice model

I am at a loss as to which R package and approach to use for my modeling. The background is: I have a transportation cross-sectional survey dataset combined with many other GIS datasets. I am ...
1
vote
0answers
14 views

Matching a query distribution to a family of template distributions

I was turning over a hypothetical question in my head: Suppose I have a set of template probability distributions, let's say each giving the probability of the occurrence of certain objects like ...
3
votes
1answer
74 views

Sample size when fitting categorical survey data

I have a model which fits data from repeated surveys: at time $t$, a number $n_t$ respondents is asked a question and can give one of $K$ answers ($k=1, ..., K$). This is repeated $T$ times ($t = 1, ...
1
vote
1answer
24 views

Model selection and assumed parameters in models

Suppose there are four models: Model 1: $y = ax$ Model 2: $y = ax^2$ Model 3: $y = a\sqrt{x}$ Model 4: $y = ax^\theta$ Model 4 is the most complex model with two parameters (the others have ...
3
votes
1answer
50 views

Help constructing a simple regression model with a breakpoint

This is related to my questions here and here. I am still struggling with my model, so I am taking it back to basics. My assertion is simple, I believe that watershed runoff will have a different ...
0
votes
0answers
20 views

Proper Model Selection Randomized Block with Count Data

I have a data set on insect counts that looks like this: ...
4
votes
1answer
37 views

How to train a model when instead of a target we have a range where it is?

Often in machine learning we have a situation when target is numeric (real or integer). Each target comes with an associated input vector. The goal is to learn the mapping from the input vectors to ...
9
votes
1answer
169 views

How to select the best fit without over-fitting data? Modelling a bimodal distribution with N normal functions, etc

I have an obviously bimodal distribution of values, which I seek to fit. The data can be fit well with either 2 normal functions (bimodal) or with 3 normal functions. Additionally, there is a ...
0
votes
1answer
19 views

Choose best samples to get strongest model

I am looking to create a model to best predict the presence of a particular species based on a number of habitat covariates. I have mapped out a couple of these covariates and am looking for a way to ...
1
vote
1answer
50 views

How to decide whether a variable belongs to a linear model?

I have a set of inputs $x$ and noisy outputs $y$. I think that either $$y = a_0 + a_1 x$$ or $$y = a_0 + a_1 x + a_2 x^2.$$ How can I determine which model was more likely to have generated the data? ...
1
vote
1answer
36 views

Does AIC require the residuals of the model to be normally distributed?

Does AIC require the residuals of the models to be compared to be normally distributed?
0
votes
0answers
24 views

Model selection (BIC / AIC) ordinal multilevel model containing a factor score and/or part of the factor

I am building a ordinal multilevel model (Stata 13.1; meologit-command). At this stage I am trying to conduct my model selection using the BIC / AIC. I estimated several models and now I need some ...
1
vote
1answer
65 views

GLM model selection using AICc with Tweedie distribution

I have two questions regarding use of Tweedie GLM in R. I am new in using this distribution and despite a thorough search on different forums, I could not find my answers. I am now running several ...
0
votes
0answers
17 views

Sampling from a set of non-nested models

Consider a collection $\mathcal{M}$ of $m$ different model classes $\mathcal{M} = \{M_1,\dots,M_m\}$, where each model class has a parameter set $\Theta_i$, $i=1,\dots,m$. The model classes are not ...
0
votes
1answer
20 views

Model selection with ncvreg

I'm quite new to R programming and was wondering, How can I read the output of the ncvreg function to find the model it selected (say with the MCP penalty)? Also how can this package be used to ...
0
votes
0answers
9 views

Removing contrast effect in mixed modelling deletion test

I am doing a mixed, linear regression analysis and now want to conduct a series of deletion tests on my max model, where I test if there is a significant increase in deviance once an effect that binds ...
1
vote
0answers
27 views

How to do regression model selection if dummy variables are involved? [duplicate]

Original post on stackoverflow: http://stackoverflow.com/questions/28773153/how-to-do-regression-model-selection-if-dummy-variables-are-involved I am trying to do a logistic regression analysis in R ...
0
votes
0answers
9 views

Text classification: Dealing with potentially false classified instances

My goal is to complete a supervised text classification task with R. There are several classes, of which some of the class counts will be relevant for a subsequent analysis. I have already tried a few ...
1
vote
0answers
24 views

Device Comparision: Correlated or uncorrelated measurements

Background: I want to compare two devices measuring a certain characteristic on a subject. Thereto, each subject is measured once with device A and once with device B. It needs to be assumed that ...
0
votes
0answers
14 views

How to apply AIC to a situation where the mean of a multivariate normal is a 0-1 d-dimensional vector with exactly k 1's

I am trying to apply AIC to estimate mean in the following case: Let us consider that I have $n$ random variables $X_1, \ldots, X_n$, drawn i.i.d. from a normal distribution of mean $\mu\in\{0,1\}^d$ ...
0
votes
0answers
51 views

Model Stacking algorithm

I'm trying the stacking method to see if it improves my results, but before using some R package, i decided to code it by myself. Here's a pseudocode of what i'm doing: ...
1
vote
1answer
70 views

Model selection: can I compare the AIC from models of count data between linear and poisson models?

I am modeling count data (with offset / exposure parameter). My modeling strategy is use of a Poisson model and a negative binomial regression model. I compare model AICs, which are about -760 for my ...