Confusion about cv.glm in R

Question

R's document says that delta is the raw cross-validation estimate of prediction error, which i think is prediction error rate in the situation of logistic regression. However, when i try to calculate prediction error rate with my own function the result is different.

cv.glm:

> fit=glm(Direction~Lag1+Lag2,family = binomial,data = Weekly)
> cv.err=cv.glm(Weekly,fit)
> cv.err$delta[1]
[1] 0.2464536

my function:

> fun=function(){
+     count=0
+     for(i in 1:length(Direction)){
+         fit=glm(Direction~Lag1+Lag2,family = binomial,data = Weekly[-i,])
+         prob=predict(fit,newdata = Weekly[i,],type = "response")
+         pred="Down"
+         if(prob>0.5)
+             pred="Up"
+         if(pred!=Direction[i])
+             count=count+1
+     }
+     return(count/length(Direction))
+ }
> fun()
[1] 0.4499541

Why the result is different? Could anyone explain this for me?

Answer 1

Logistic regression is not a classification algorithm, and the decision rule you used (i.e. prob > 0.5 cutoff) is not a part of logistic regression model.

Logistic regression predicts conditional probabilities of successes, so you should instead calculate your errors accordingly, i.e. comparing to the probabilities, not to the probabilities rounded to {0, 1}.

Below you can see an example where errors from leave-one-out cross-validation are calculated for model using your decision rule, and the ordinary errors for logistic regression. As you can see, cv.glm uses the second approach.

fit <- glm(vs ~ mpg, data = mtcars, family = binomial)
out <- NULL
for (i in 1:nrow(mtcars))
  out[i] <- predict(update(fit, data = mtcars[-i,]), newdata = mtcars[i,], type = "response")

boot::cv.glm(mtcars, fit)$delta[1]
## 0.1497903
mean((mtcars$vs - round(out))^2)
## 0.1875
mean((mtcars$vs - out)^2)
## 0.1497903

Answer 2

Leaving this as an answer instead of a simple comment because I don't have the reputation to comment. The question asks, why is it different, and the accepted answer is excellent. The one piece missing was how to get the correct answer from cv.glm().

Using the help function ?cv.glm the last example is using cv.glm get error rate on binary prediction model. Taking the cost function from there and inserting into your example:

fit=glm(Direction~Lag1+Lag2,family = binomial,data = Weekly)
cost <- function(r, pi) mean(abs(r-pi)> 0.5)
cv.err=cv.glm(Weekly,fit, cost = cost)
cv.err$delta[1]
[1] 0.4499541

You get the same error as your manual example.