1
$\begingroup$

Looking at the cv.glm function I am trying to understand how its cost function parameter is used.

I think I understand that cost gets called to calculate the "error" for the test set. But it gets called several more times than I have expected. Take a look at the following code using the data sets mentioned in the Introduction to Statistical Learning book.

library(ISLR)
library(boot)
set.seed(1)

nrow(Auto)
# prints: [1] 392 

cost <- function(r, pi) {
  print("Calling into costFunction")
  print(sprintf("r lenght: %s", length(r)))
  # print(r)
  print(sprintf("pi lenght: %s", length(pi)))
  # print(pi)
  return(mean((r - pi)^2))
}

glm.fit <- glm(mpg ~ horsepower, data = Auto)
cv.err <- cv.glm(Auto, glm.fit, cost = cost, K = 2)
cv.err$delta

Here I have a cost function that still returns the mean squared error with some debug prints. The number of datapoints in Auto data is 392. I am doing K-fold cross validation where K is 2.

Running this prints

[1] "Calling into costFunction"
[1] "r lenght: 392"
[1] "pi lenght: 392"
[1] "Calling into costFunction"
[1] "r lenght: 196"
[1] "pi lenght: 196"
[1] "Calling into costFunction"
[1] "r lenght: 392"
[1] "pi lenght: 392"
[1] "Calling into costFunction"
[1] "r lenght: 196"
[1] "pi lenght: 196"
[1] "Calling into costFunction"
[1] "r lenght: 392"
[1] "pi lenght: 392"

With K = 2, the test sets would be 392/2 = 196. So the calls to cost with input arguments with 196 entries make sense. There are two such folds and each having half of the original data (=196 rows) as the test set.
But what are the other calls to cost function with 392 long vectors? Total number of data points is 392.

$\endgroup$
1
$\begingroup$

The source code of cv.glm shows the call places for the cost function.

The other calls to cost are used to calculate the bias corrected MSE. In the docs for cv.glm it mentions as:

delta

A vector of length two. The first component is the raw cross-validation estimate of prediction error. The second component is the adjusted cross-validation estimate. The adjustment is designed to compensate for the bias introduced by not using leave-one-out cross-validation.

$\endgroup$
1
  • 1
    $\begingroup$ cool. was about to answer the question, but you found it $\endgroup$
    – StupidWolf
    Jun 21 '20 at 22:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.