R's document says that delta is the raw cross-validation estimate of prediction error, which i think is prediction error rate in the situation of logistic regression. However, when i try to calculate prediction error rate with my own function the result is different.


> fit=glm(Direction~Lag1+Lag2,family = binomial,data = Weekly)
> cv.err=cv.glm(Weekly,fit)
> cv.err$delta[1]
[1] 0.2464536

my function:

> fun=function(){
+     count=0
+     for(i in 1:length(Direction)){
+         fit=glm(Direction~Lag1+Lag2,family = binomial,data = Weekly[-i,])
+         prob=predict(fit,newdata = Weekly[i,],type = "response")
+         pred="Down"
+         if(prob>0.5)
+             pred="Up"
+         if(pred!=Direction[i])
+             count=count+1
+     }
+     return(count/length(Direction))
+ }
> fun()
[1] 0.4499541

Why the result is different? Could anyone explain this for me?

  • 2
    $\begingroup$ One simple reason why your result differs is that you assumed that prob>0.5 is labeled as 1, while glm does not make such assumptions, but simply predicts the probabilities of success. Compute your errors from what glm predicted not from rounded predictions. $\endgroup$ – Tim Dec 10 '16 at 18:24

Logistic regression is not a classification algorithm, and the decision rule you used (i.e. prob > 0.5 cutoff) is not a part of logistic regression model.

Logistic regression predicts conditional probabilities of successes, so you should instead calculate your errors accordingly, i.e. comparing to the probabilities, not to the probabilities rounded to {0, 1}.

Below you can see an example where errors from leave-one-out cross-validation are calculated for model using your decision rule, and the ordinary errors for logistic regression. As you can see, cv.glm uses the second approach.

fit <- glm(vs ~ mpg, data = mtcars, family = binomial)
out <- NULL
for (i in 1:nrow(mtcars))
  out[i] <- predict(update(fit, data = mtcars[-i,]), newdata = mtcars[i,], type = "response")

boot::cv.glm(mtcars, fit)$delta[1]
## 0.1497903
mean((mtcars$vs - round(out))^2)
## 0.1875
mean((mtcars$vs - out)^2)
## 0.1497903
  • 3
    $\begingroup$ +1. I would also add that, at least with logistic regression, it would be a good idea to pass in a cost argument to gv.glm, so that the cross validation is minimizing the logistic loss. $\endgroup$ – Matthew Drury Dec 10 '16 at 19:07

Leaving this as an answer instead of a simple comment because I don't have the reputation to comment. The question asks, why is it different, and the accepted answer is excellent. The one piece missing was how to get the correct answer from cv.glm().

Using the help function ?cv.glm the last example is using cv.glm get error rate on binary prediction model. Taking the cost function from there and inserting into your example:

fit=glm(Direction~Lag1+Lag2,family = binomial,data = Weekly)
cost <- function(r, pi) mean(abs(r-pi)> 0.5)
cv.err=cv.glm(Weekly,fit, cost = cost)
[1] 0.4499541

You get the same error as your manual example.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.