10
$\begingroup$

Given multiple cross-validation folds of a logistic regression, and the resulting multiple estimates of each regression coefficient, how should one measure whether or not a predictor (or set of predictors) is/are stable and meaningful based on the regression coefficient(s)? Is this different for linear regression?

$\endgroup$
3
  • $\begingroup$ @BGreene Very clever. Why not post that as an answer? You're also making me wonder whether the ensemble learning literature holds something relevant. $\endgroup$ Feb 28, 2013 at 17:14
  • $\begingroup$ When you say "multiple cross-validation", do you mean that you run $m$ times a $k$-fold cross-validation? $\endgroup$
    – boscovich
    Mar 11, 2013 at 9:22
  • $\begingroup$ @andrea, I say "multiple cross-validation folds", i.e., $k$ folds. $\endgroup$ Mar 11, 2013 at 19:48

2 Answers 2

2
$\begingroup$

You could treat the regression coefficients resulting from each test fold in the CV as independent observations and then calculate their reliability/stability using intra-class correlation coefficient (ICC) as reported by Shrout & Fleiss.

$\endgroup$
0
$\begingroup$

I assume you in your cross-validation you divide the data in two parts, a training set and a test set. In one fold you fit a model from the training set and use it to predict the response of the test set, right? This will give you an error rate for the whole model, not for a single predictor.

I do not know if it is possible to find p-values for predictors using something like the F-tests used in ordinary linear regression.

You can try remove predictors from the model using for example backward or forward selection if that is your aim.

You could instead of CV use bootstrap to find a confidence interval for each predictor and then see how stable it is.

How many folds do you use in your CV, is it leave-one-out cross-validation?

Perhaps more details of what your aim is would help to answer this question.

$\endgroup$
1
  • $\begingroup$ Assume this is leave-one-out. Each predictor in each fold already has a confidence interval, e.g., from a Bayesian posterior CI or std err from glm(..., family="binomial") in R. What do I do with the intervals for each predictor across the leave-one-out runs? $\endgroup$ Apr 27, 2012 at 3:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.