I'm ultimately looking to apply cross-validation to a much more complicated scenario, but for now consider the simple task of leave-one-out-cross-validating logistic regression. When you compare the left-out value against its respective model's predicted value, I presume you must convert the predicted value to the probability scale, but then do you also round the predicted value so that its either 0 or 1, or leave it unrounded? Any ideas on which is the better approach (and maybe why)?

Example data and by-hand loocv of a logistic model:

myData = data.frame(
 x = seq(-2,2,length.out = 10)
myData$y = rbinom(nrow(myData),1,plogis(myData$x))

myData$preds = NA
for(i in 1:nrow(myData)){
    thisFit = glm(
        data = myData[(1:nrow(myData)!=i),]
        , formula = y~x
        , family = binomial
    myData$preds[i] = predict(thisFit,newdata=myData[(1:nrow(myData)==i),])

myData$predsProb = plogis(myData$preds)

#one approach:
myData$squaredError1 = (myData$y - myData$predsProb)^2
SSE1 = sum(myData$squaredError1)

myData$squaredError2 = (myData$y - round(myData$predsProb))^2
SSE2 = sum(myData$squaredError2)

#which is a "better" estimate of future prediction error, SSE1 or SSE2? Why?

It would be wise to study proper scoring rules, which will score continuous risk predictions. Also, make sure you want to do (expensive) leave-out-one cv instead of bootstrap or averaging multiple runs of 10-fold cv.

Take a look at the following functions in the R rms package: lrm, validate.lrm, calibrate.lrm, val.prob (the latter is for external validation but provides a larger set of accuracy scores).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.