Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 8 down vote favorite
2

Given multiple cross-validation folds of a logistic regression, and the resulting multiple estimates of each regression coefficient, how should one measure whether or not a predictor (or set of predictors) is/are stable and meaningful based on the regression coefficient(s)? Is this different for linear regression?

share|improve this question
@BGreene Very clever. Why not post that as an answer? You're also making me wonder whether the ensemble learning literature holds something relevant. – Jack Tanner Feb 28 at 17:14
thanks, will do. – BGreene Feb 28 at 18:48
When you say "multiple cross-validation", do you mean that you run $m$ times a $k$-fold cross-validation? – andrea Mar 11 at 9:22
@andrea, I say "multiple cross-validation folds", i.e., $k$ folds. – Jack Tanner Mar 11 at 19:48

2 Answers

up vote 1 down vote

You could treat the regression coefficients resulting from each test fold in the CV as independent observations and then calculate their reliability/stability using intra-class correlation coefficient (ICC) as reported by Shrout & Fleiss.

share|improve this answer
up vote 0 down vote

I assume you in your cross-validation you divide the data in two parts, a training set and a test set. In one fold you fit a model from the training set and use it to predict the response of the test set, right? This will give you an error rate for the whole model, not for a single predictor.

I do not know if it is possible to find p-values for predictors using something like the F-tests used in ordinary linear regression.

You can try remove predictors from the model using for example backward or forward selection if that is your aim.

You could instead of CV use bootstrap to find a confidence interval for each predictor and then see how stable it is.

How many folds do you use in your CV, is it leave-one-out cross-validation?

Perhaps more details of what your aim is would help to answer this question.

share|improve this answer
Assume this is leave-one-out. Each predictor in each fold already has a confidence interval, e.g., from a Bayesian posterior CI or std err from glm(..., family="binomial") in R. What do I do with the intervals for each predictor across the leave-one-out runs? – Jack Tanner Apr 27 '12 at 3:22

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.