Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

learn more… | top users | synonyms (1)

1
vote
1answer
16 views

Duplicate AICc values for multiple models with interactions

I am going through a model selection process with a mixed-model with 3 variables: A, B, and C. B and C are orthogonal factors. B or C may interact with A, so my full model would be: fixed: ...
2
votes
0answers
33 views

ARX model selection

I have an autoregressive model with exogenous variables: $S_{t} = \sum_{i=1}^{p} a_i S_{t-i} + \sum_q \sum_{i=1}^{r} b^q_i X^{q}_{t-i}$ where $S_t$ is the signal I want to predict and $X^q_t$ the ...
1
vote
0answers
3 views

Model comparison using different measures for the independent variables

I am doing a scale development study and would like to compare the newly developed measurement scale with an existing measure for the same construct and determine which scale better explains a set of ...
2
votes
0answers
12 views

The correct use of tensor product in gam (mgcv) function

I want to resurrect a question that I asked two months ago (Comparing gam models using ti( )), but adding more explanations. The aim of my analyses was to compare several gam models with different ...
3
votes
1answer
118 views

How to argue omitted variable problem is alleviated?

Is there any ways to argue that the omitted variable problem is alleviated after adding a new variable to the model? Right now I'm basically just saying that adding this new variable significantly ...
2
votes
1answer
58 views

Can you compare different functional models using Akaike criterion?

I have two regression models. One is a simple linear regression model ($y$ is regressed on $x_i$'s), while the other is a double log model (log of $y$ is regressed on log of the $x_i$'s). They have ...
0
votes
1answer
78 views

Fit data to a bivariate function

I want to fit my (x,y,z) data points to a function. You can see the data on Fig.1. The data is symmetric along the main diagonal. To understand my data I have studied (y,z) curves at different ...
1
vote
1answer
45 views

Model selection and comparison in GAMM using R (mgcv)

I'm fitting a GAMM with correlation structure, using a non-Gaussian family. Here's an example of my global model: ...
0
votes
0answers
8 views

Should information crtieria be applied to training or validation data?

Information criteria for selecting models seem to be applied to training data in general. Could they also be applied to validation data to decide the most predictive and simple model, or is this ...
0
votes
1answer
14 views

What are comments about the model specification when significance levels decrease in the multivariate model?

So I run the multivariate model and the significance levels decrease (i.e., larger p values) for almost all predictors. The extent to which they decrease vary, some may go from <0.001 to ...
0
votes
0answers
19 views

negative AIC or positive AIC? [duplicate]

I have calculated AIC by using R-studio to compare models. However, I got the following both negative and positive AIC AIC 8.52 0.41 -7.70 -5.84 -3.84 -2.10 Should I select the negative AIC or ...
1
vote
0answers
21 views

Help with R packages to use in my spatial discrete choice model

I am at a loss as to which R package and approach to use for my modeling. The background is: I have a transportation cross-sectional survey dataset combined with many other GIS datasets. I am ...
1
vote
0answers
14 views

Matching a query distribution to a family of template distributions

I was turning over a hypothetical question in my head: Suppose I have a set of template probability distributions, let's say each giving the probability of the occurrence of certain objects like ...
3
votes
1answer
68 views

Sample size when fitting categorical survey data

I have a model which fits data from repeated surveys: at time $t$, a number $n_t$ respondents is asked a question and can give one of $K$ answers ($k=1, ..., K$). This is repeated $T$ times ($t = 1, ...
1
vote
1answer
17 views

Model selection and assumed parameters in models

Suppose there are four models: Model 1: $y = ax$ Model 2: $y = ax^2$ Model 3: $y = a\sqrt{x}$ Model 4: $y = ax^\theta$ Model 4 is the most complex model with two parameters (the others have ...
3
votes
1answer
39 views

Help constructing a simple regression model with a breakpoint

This is related to my questions here and here. I am still struggling with my model, so I am taking it back to basics. My assertion is simple, I believe that watershed runoff will have a different ...
0
votes
0answers
19 views

Proper Model Selection Randomized Block with Count Data

I have a data set on insect counts that looks like this: ...
4
votes
1answer
33 views

How to train a model when instead of a target we have a range where it is?

Often in machine learning we have a situation when target is numeric (real or integer). Each target comes with an associated input vector. The goal is to learn the mapping from the input vectors to ...
9
votes
1answer
144 views

How to select the best fit without over-fitting data? Modelling a bimodal distribution with N normal functions, etc

I have an obviously bimodal distribution of values, which I seek to fit. The data can be fit well with either 2 normal functions (bimodal) or with 3 normal functions. Additionally, there is a ...
0
votes
1answer
18 views

Choose best samples to get strongest model

I am looking to create a model to best predict the presence of a particular species based on a number of habitat covariates. I have mapped out a couple of these covariates and am looking for a way to ...
1
vote
1answer
46 views

How to decide whether a variable belongs to a linear model?

I have a set of inputs $x$ and noisy outputs $y$. I think that either $$y = a_0 + a_1 x$$ or $$y = a_0 + a_1 x + a_2 x^2.$$ How can I determine which model was more likely to have generated the data? ...
1
vote
1answer
32 views

Does AIC require the residuals of the model to be normally distributed?

Does AIC require the residuals of the models to be compared to be normally distributed?
0
votes
0answers
21 views

Model selection (BIC / AIC) ordinal multilevel model containing a factor score and/or part of the factor

I am building a ordinal multilevel model (Stata 13.1; meologit-command). At this stage I am trying to conduct my model selection using the BIC / AIC. I estimated several models and now I need some ...
1
vote
1answer
51 views

GLM model selection using AICc with Tweedie distribution

I have two questions regarding use of Tweedie GLM in R. I am new in using this distribution and despite a thorough search on different forums, I could not find my answers. I am now running several ...
0
votes
0answers
13 views

Sampling from a set of non-nested models

Consider a collection $\mathcal{M}$ of $m$ different model classes $\mathcal{M} = \{M_1,\dots,M_m\}$, where each model class has a parameter set $\Theta_i$, $i=1,\dots,m$. The model classes are not ...
0
votes
1answer
18 views

Model selection with ncvreg

I'm quite new to R programming and was wondering, How can I read the output of the ncvreg function to find the model it selected (say with the MCP penalty)? Also how can this package be used to ...
0
votes
0answers
6 views

Removing contrast effect in mixed modelling deletion test

I am doing a mixed, linear regression analysis and now want to conduct a series of deletion tests on my max model, where I test if there is a significant increase in deviance once an effect that binds ...
1
vote
0answers
25 views

How to do regression model selection if dummy variables are involved? [duplicate]

Original post on stackoverflow: http://stackoverflow.com/questions/28773153/how-to-do-regression-model-selection-if-dummy-variables-are-involved I am trying to do a logistic regression analysis in R ...
0
votes
0answers
8 views

Text classification: Dealing with potentially false classified instances

My goal is to complete a supervised text classification task with R. There are several classes, of which some of the class counts will be relevant for a subsequent analysis. I have already tried a few ...
1
vote
0answers
24 views

Device Comparision: Correlated or uncorrelated measurements

Background: I want to compare two devices measuring a certain characteristic on a subject. Thereto, each subject is measured once with device A and once with device B. It needs to be assumed that ...
0
votes
0answers
14 views

How to apply AIC to a situation where the mean of a multivariate normal is a 0-1 d-dimensional vector with exactly k 1's

I am trying to apply AIC to estimate mean in the following case: Let us consider that I have $n$ random variables $X_1, \ldots, X_n$, drawn i.i.d. from a normal distribution of mean $\mu\in\{0,1\}^d$ ...
0
votes
0answers
35 views

Model Stacking algorithm

I'm trying the stacking method to see if it improves my results, but before using some R package, i decided to code it by myself. Here's a pseudocode of what i'm doing: ...
1
vote
1answer
52 views

Model selection: can I compare the AIC from models of count data between linear and poisson models?

I am modeling count data (with offset / exposure parameter). My modeling strategy is use of a Poisson model and a negative binomial regression model. I compare model AICs, which are about -760 for my ...
0
votes
0answers
57 views

AIC versus cross validation in time series

I am interested in model selection in a time series setting. For concreteness, suppose I want to select an ARMA model from a pool of ARMA models with different lag orders. The ultimate intent is ...
0
votes
0answers
30 views

Heckman Probit Model the number of explanatory variable in selection model?

I run a Heckman Probit model which is sometimes called as Heckit. It consists two parts like this: |1| Y X1 X2 X3, |2| select(Y2 X1 X2 X3) Y covers Y2 but not vice versa. The question is whether i ...
7
votes
2answers
207 views

Is using correlation matrix to select predictors for regression correct?

A few days ago, a psychologist-researcher of mine told me about his method to select variables to linear regression model. I guess it's not good, but I need to ask someone else to make sure. The ...
1
vote
0answers
47 views

Strictly positive response in regression: what should my “default” model be?

For unbounded continuous responses, Gaussian errors are the analyst's default model for many reasons, one of them being that their ML estimate coincides with the OLS estimate that has many desirable ...
1
vote
1answer
37 views

Proper variable selection: Use only training data or full data?

I'm going through the lab exercises in "Introduction to Statistical Learning" and am having difficulty understanding the proper way to do best subset selection. The book is available here ...
0
votes
0answers
15 views

VAR model selection for forecasting one variable

Suppose I have a VAR model for variables $x_1$ through $x_K$. I will use the model to forecast $x_1$ a few steps ahead and will do this iteratively rather than directly. I am not interested in ...
1
vote
0answers
17 views

Are wrong standard errors a problem if using information theoretic model selection?

In linear regression, if the assumptions of normally distributed residuals and homogenous residuals are broken, incorrect standard errors can be calculated. This can lead to some predictors appearing ...
0
votes
0answers
22 views

Model choice for discrete data with R

I'm keen to model the effect (impact) of variable i.e. var1 and its effect (causation inference?) on var2 given the score measure. Sample data (variation of data I have got) ...
0
votes
0answers
9 views

ensemble model for SVM

I did a nested 5-cv and the resulting models are unstable (high variance among the hyper parameters C and gamma of SVM). So, I don't know how to choose C and gamma for the "final" model. I read that, ...
1
vote
1answer
39 views

Goodness of fit between actual values and non-linear model

I was wanting to get a goodness of fit similar to R^2 for a model I'm evaluating. The output of the model is one of 8 numbers based on environmental characteristics. This is not a linear model, so ...
1
vote
0answers
56 views

GLMM with 2 insignificant variables has lower AIC or BIC compared to same model without those variables…?

I am having a hard time understanding what's going on in with my model selection, and why a model with two insignificant variables is getting chosen as the "best model" over a model without those two ...
2
votes
0answers
56 views

Model selection

I have a small dataset of 37 observations with students' performance on both cognitive tests (5) and professional tests (6). My goal is to predict professional tests (DV) with cognitive tests(IV). To ...
1
vote
0answers
33 views

How to deal with many NAs in dataset when comparing several models?

My task is to compare different logistic regression models to examine which theory better explains the data we have collected. In doing so, I have a set of IVs with a considerable amount of missing ...
0
votes
0answers
22 views

Model Selection Problem

I am asking if there already exist approaches and researches on the following topic. Imagine there are 10 stores and in 3 stores labeled training data was available, so I built 3 classification ...
0
votes
1answer
42 views

Comparison between a multilevel and an unpooled model

Suppose we have fitted two models: a multilevel model and an unpooled model: m1=lmer(y~x+(1|group)) m2=lm(y~x+factor(group)-1) How can I understand which ...
0
votes
1answer
74 views

How to identify the ARCH and GARCH lag lenght in DCC GARCH Model

I just follow the Stata manual for DCC GARCH model. This model contains ARCH(1) and GARCH(1) terms. But my question is, on what basis and how we can can select ...
0
votes
1answer
90 views

Leave-One-Subject-Out cv method

I would like to use a Leave-One-Subject-Out cv on my datasets (I have dataset including 38, 15, 10 participants, respectively). I don't know the hyperparamenters C and gamma of my SVM so I have to ...