Questions tagged [model-selection]

Model selection is a problem of judging which model from some set performs best. Popular methods include $R^2$, AIC and BIC criteria, test sets, and cross-validation. To some extent, feature selection is a subproblem of model selection.

Filter by
Sorted by
Tagged with
5 votes
2 answers
988 views

Bias in classifier model selection

Say I have a set of classifier models, each generated using feature selection inside a repeated k-fold cross-validation. Each classifier model is generated using a different set of regularization ...
user avatar
  • 3,125
3 votes
2 answers
681 views

Is it OK to fit random effects before fixed effects?

I'm using lmer to analyse my data, building nested models and using anova() to compare them against each other in an incremental way. Now, I know enough to only test a single term at a time (i.e. only ...
user avatar
  • 71
1 vote
0 answers
97 views

Evaluation of groups of parameters across multiple data sets in logistic regression [closed]

I have a logistic regression problem where I want to evaluate the aggregate contribution of groups of parameters (independent variables) across different sets of observations. For instance, let's say ...
user avatar
  • 11
5 votes
0 answers
653 views

AIC with Mantel's tests

Mantel's tests are commonly used to compare genetic distances (say, between a number of individuals) with true or hypothesized landscape distances between those same individuals. For example, “does ...
user avatar
2 votes
0 answers
60 views

Analysing choices pattern

we have a process in which, at each step, a set of elements are presented to user, the user choses one, his choice is recorded and next round starts with a new set of elements. For example: 1. {20,50,...
user avatar
  • 21
4 votes
2 answers
5k views

Following up a three-way interaction found with (mixed) model comparison

I hope this isn't a silly question, I'd like some advice on following up a threeway interaction in a mixed effects model. I've been building my models incrementally, like this: ...
user avatar
  • 71
6 votes
2 answers
1k views

Gaussian Process goodness of fit

Let's say I got a Gaussian Process model $M$ based on some training data. Now I get a stream of sample data of a certain batch size coming in. The GP does not model a time series, but it's trying to ...
user avatar
2 votes
1 answer
317 views

How to specify and estimate the parameters of a model that is quadratic in several variables

I am trying to find how the quadratic model for a multiple featured dataset will be. Suppose that my training set is $X_1,...X_n$ with each $X$ of dimension $4$. Now suppose I want to fit a quadratic ...
user avatar
  • 1,381
8 votes
1 answer
3k views

Model-selection for linear mixed models over alternative sets of parameters (nlme function in R)

My models look like: ...
user avatar
  • 283
0 votes
3 answers
320 views

Factor significant within model but non significant after drop?

This may be quite a basic question but I was running a simple linear model and dropping non significant terms until I got to a minimal model. When this was reached, I was obtaining the significance ...
user avatar
  • 179
0 votes
1 answer
324 views

Understanding what Dassault iSight is doing?

I've been tasked with taking up some modeling (perhaps predictive) for chemical formulations and some resultant performance characteristics. I have experience with finite element analysis (mechanical ...
user avatar
  • 281
22 votes
3 answers
4k views

Model stability when dealing with large $p$, small $n$ problem

Intro: I have a dataset with a classical "large p, small n problem". The number available samples n=150 while the number of possible predictors p=400. The outcome is a continuous variable. I want ...
user avatar
  • 321
7 votes
1 answer
408 views

Multivariate time series model evaluation with conditional moments

Consider multivariate time series models that estimate potentially time-varying conditional means, variances, and correlations (one type of model might be a VAR(p)+Garch(1,1)+DCC Gaussian Copula model)...
user avatar
  • 2,137
2 votes
1 answer
3k views

Regression technique for data comprised of categorical explanatory variables & a continuous response variable

i suppose one way to characterize data is by a combination of the variable types that comprises it: ...
user avatar
  • 10.1k
2 votes
1 answer
377 views

Using mle2() for age-period-cohort models

I've followed guidelines for comparing models in Chapter 6 of Bolker's Ecological Models and Data in R, applying code used in this section to cancer count data. The models include parameters for age ...
user avatar
1 vote
3 answers
3k views

Can eliminating parameters reduce overfitting?

I am learning a statistical model, which includes a very large amount of parameters, which results in the risk of over-fitting. If I first learn the model parameters from the data, and then simply ...
user avatar
  • 129
-4 votes
1 answer
2k views

Best way to compare two alternative models

Let's suppose we are attempting to predict traffic accidents on some highway. Traffic Accidents would be the response variable in our data set. We have TWO competing models: Two predictor variables:...
user avatar
  • 141
-14 votes
1 answer
2k views

Finding the best two predictor variables used conjointly, and levels of each [closed]

Twenty possible predictor variables in data set. One outcome variable. Some of the predictor variables are not linear. So a standard linear multiple regression approach probably won't do. (And I do ...
user avatar
  • 141
3 votes
0 answers
2k views

Are very large log likelihood and delta AIC values problematic for model selection?

I am using AICc for small sample sizes to compare 8 a priori models (including null model). I fitted my models using a GLMM due to the nested nature of my data and defined the family as 'poisson' ...
user avatar
  • 31
1 vote
0 answers
232 views

How to compare the performence between different models?

Given a data, I'd like to utilize it by two different models, such as generalized linear model (GLM) and generalized partial linear model (GPLM), and my problem is how to compare the performance of ...
user avatar
7 votes
1 answer
1k views

Model selection in a paper: what to say about the dropped variables?

I have a question (simpler than my previous post today I hope!), which is probably very stupid as nobody has never asked it before. Lets say I am trying to explain the effect of 3 variables (A, B ...
user avatar
  • 195
17 votes
2 answers
7k views

What are chunk tests?

In answer to a question on model selection in the presence of multicollinearity, Frank Harrell suggested: Put all variables in the model but do not test for the effect of one variable adjusted for ...
user avatar
  • 4,767
1 vote
0 answers
132 views

Calculate the number of trials to distinguish between 2 models

This sounds like a problem that has to be well known, but I can't find a good answer. I have tried another forum before I realized there was a stackexchange stats site. So here goes ... I have two ...
user avatar
  • 11
11 votes
1 answer
1k views

ABC model selection

It has been shown that ABC model choice using Bayes factors is not to be recommended due to the presence of an error coming from the use of summary statistics. The conclusion in this paper relies on ...
user avatar
3 votes
1 answer
840 views

Why does -2*LL differ when using binary logistic regression vs GLM binary logistic in SPSS?

I'm comparing plausible models selected a-priori to predict a binary response variable. I used binary logistic regression in SPSS20 and obtained AIC=-2*LogLikelihood+2k where k is the number of ...
user avatar
  • 73
3 votes
1 answer
133 views

Thoughts on model self-penalization amidst difficult parameter estimation

It is well accepted that one should account for model complexity when performing model comparisons, and the general procedure is to penalize more complex models more strongly. While this makes sense ...
user avatar
2 votes
0 answers
2k views

How do I determine which functional form is correct?

I have an assignment that asks me to explore different series of data with respect to the effects that an incinerator built in 1981 in Massachusetts, has on the price of houses in that area. I have ...
user avatar
  • 21
36 votes
3 answers
20k views

Is it possible to calculate AIC and BIC for lasso regression models?

Is it possible to calculate AIC or BIC values for lasso regression models and other regularized models where parameters are only partially entering the equation. How does one determine the degrees of ...
user avatar
  • 864
0 votes
0 answers
212 views

Computing probability of proportions of (possibly) overlapping sets

Consider $n$ fixed subsets of a finite universe $U$: $A,B,C,D,\ldots$ Then let $Coll$ be the collection of sets we get from the closure of the set $\{U,A,B,C,D, \ldots\}$ under the operations $\cup,\...
user avatar
2 votes
2 answers
4k views

Relative variable importance values vs. magnitude of effect

I have ran a series of models to see which best fit the response variable and I got the following (for the model average of all models with a $\Delta AIC < 2$). I am currently learning models so ...
user avatar
2 votes
0 answers
133 views

How to learn fitting models?

I am new to R(learned the basics) In my current job I have to solve the following problem(the following is an example) Suppose we have 10, 000 users For each user we have his expected ...
user avatar
  • 119
4 votes
2 answers
970 views

52 variables after backward variable selection on logistic regression on 160 variable at beginning, whether it is illusion or good modeling

Thank you for you guys effort on building such a nice community and I learned lot by reading and asking. Background about my logistic regression: Target Variable: Binary target about volatility of ...
user avatar
1 vote
0 answers
154 views

Identification of significant variables where dependent variable is categorical

I have a set of some 50,000 data points. There is one dependent variable which is categorical in nature and there are some 100 possible explanatory variables. Out of these 100, I have to select some ...
user avatar
1 vote
2 answers
589 views

Why were results from PROC MIXED same as PROC GLIMMIX?

I am running a simple code comparing insect abundance on a vegetated and unvegetated surfaces. Design is RCBD. 1 factor with 2 level (vegetated and unvegetated). 10 blocks. Distribution of residuals ...
user avatar
  • 11
19 votes
5 answers
14k views

Can I ignore coefficients for non-significant levels of factors in a linear model?

After seeking clarification about linear model coefficients over here I have a follow up question concerning non-signficant (high p value) for coefficients of factor levels. Example: If my linear ...
user avatar
10 votes
2 answers
3k views

Model stability in cross-validation of regression models

Given multiple cross-validation folds of a logistic regression, and the resulting multiple estimates of each regression coefficient, how should one measure whether or not a predictor (or set of ...
user avatar
  • 4,602
6 votes
4 answers
2k views

Communicating Regression Model Results

I am concerned about how unequipped most people are (both within and without academia) to properly employ standard model building methods such as linear regression and to interpret the results of ...
user avatar
  • 71
2 votes
2 answers
983 views

Model averaging in prediction -- "Wisdom of the Crowd"

Suppose I'm trying to predict $Y$ (a real number) and I have $n$ experts with guesses $Y_1,...Y_n$. Each prediction is a reasonable guess as to the value of Y in itself (hence the name "expert"), but ...
user avatar
  • 213
2 votes
1 answer
211 views

Help with multilinear regression model selection

The model is from sports. I'm trying to predict the win percentage of each team in the next season based on information available before that season. I have about 240 observations of the numbers of ...
user avatar
  • 213
7 votes
0 answers
162 views

Graphical nominal model

Suppose I have a set of $k$ matrices. $$ \mathbb D = A_1,A_2,...,A_k $$ Each column of $A$ is categorical vector. $$ A = v_1,v_2,...,v_n $$ I want to find the mapping $$ f: A \...
user avatar
7 votes
1 answer
2k views

How to select the final model with elastic net feature selection, cross validation and SVM?

I have a dataset of some 100 samples, each with >10,000 features, some of which highly correlated. Here's what I am doing currently. Split the data set into three folds. For each fold, 2.1 Run ...
user avatar
6 votes
1 answer
395 views

How to compare weighted multivariate linear models?

I've got a set of multivariate regression models, with weights, that I'm trying to compare in R. Looks like: ...
user avatar
  • 822
2 votes
2 answers
2k views

Interaction term as a dependant variable in LMM with R

In a longitudinal study, two groups of subjects have been measured over a period of two years at 6 months intervals. During these measurements subjects have been assessed with a series of $k$ measures ...
user avatar
  • 985
1 vote
1 answer
73 views

Estimating a linear function mapping from n-dim to 1-dim [closed]

I'm coming from a computer science domain with an estimation problem that I'm trying to address. My background isn't statistics, so I apologize if the terminology is bad. I've been reading various ...
user avatar
2 votes
1 answer
131 views

Statistical model for my problem

I'm running a benchmark to find the efficiency of a my computer. There are $p$ control variables say, $x_1,x_2,...,x_p$ and one output variable $Y$. For example, every time I run an experiment I ...
user avatar
  • 121
2 votes
0 answers
1k views

Using autocorrelation plots to choose the number of inputs for a neural network predicting time series

A neural network applied to time series needs to have the number of input nodes defined. Each input is applied to a time point previous to the current point being predicted. If $D$ is the number of ...
user avatar
  • 1,485
7 votes
1 answer
4k views

Subtree replacement vs subtree raising

As we saw in this question, the recommended strategy of building a decision tree is postpruning. The two methods for that are subtree replacement and subtree raising. At each node, an algorithm ...
user avatar
  • 3,277
7 votes
3 answers
9k views

Model evaluation and comparison for selecting the best model

When comparing results obtained with different models in R, what should I look for to select the best one? If I use for example the following 4 models applied to the same presence/absence sample ...
user avatar
27 votes
1 answer
2k views

Appropriate residual degrees of freedom after dropping terms from a model

I am reflecting on the discussion around this question and particularly Frank Harrell's comment that the estimate for variance in a reduced model (ie one from which a number of explanatory variables ...
user avatar
  • 16.7k
15 votes
2 answers
945 views

LASSO/LARS vs general to specific (GETS) method

I have been wondering, why are LASSO and LARS model selection methods so popular even though they are basically just variations of step-wise forward selection (and thus suffer from path dependency)? ...
user avatar
  • 151