14
$\begingroup$

I'm doing a cross validation using the leave-one-out method. I have a binary response and am using the boot package for R, and the cv.glm function. My problem is that I don't fully understand the "cost" part in this function. From what I can understand this is the function that decides whether an estimated value should be classified as a 1 or a 0, i.e the threshold value for the classification. Is this correct?

And, in the help in R they use this function for a binomial model: cost <- function(r, pi = 0) mean(abs(r-pi) > 0.5). How do I interpret this function? so I can modify it correctly for my analysis.

Any help is appreciated, don't want to use a function I don't understand.

$\endgroup$
9
$\begingroup$

r is a vector that contains the actual outcome, pi is a vector that contains the fitted values.

cost <- function(r, pi = 0) mean(abs(r-pi) > 0.5)

This is saying $cost = \sum|r_i - pi_i|$. You can define your own cost functions. In your case for binary classification you can do something like this

mycost <- function(r, pi){
    weight1 = 1 #cost for getting 1 wrong
    weight0 = 1 #cost for getting 0 wrong
    c1 = (r==1)&(pi<0.5) #logical vector - true if actual 1 but predict 0
    c0 = (r==0)&(pi>=0.5) #logical vector - true if actual 0 but predict 1
    return(mean(weight1*c1+weight0*c0))
  }

and put mycost as an argument in the cv.glm function.

$\endgroup$
  • $\begingroup$ Shouldn't $cost$ be something like $$\sum \Bigl\lfloor \frac{|r_i-p_i|}{0.5}\Bigr\rfloor$$ (with the exception, when $|r_i-p_i|=1$, then the term is $1$, not $2$)? $\endgroup$ – Mooncrater Aug 8 '17 at 16:06
  • $\begingroup$ @feng-mai pi==0 or pi < 0.5? ( and pi==1 or pi > 0.5?) if using 0.5 as the decision boundary. Are not the pi the predicted probabilities? $\endgroup$ – PM. Jun 4 '18 at 9:33
  • 1
    $\begingroup$ @PM Yes you are right. $pi$ are the responses from the glm model. Thanks for the correction. $\endgroup$ – Feng Mai Jun 10 '18 at 17:26
1
$\begingroup$
cost <- function(r, pi = 0) mean(abs(r-pi) > 0.5)

First, you have set a cut-off as 0.5. Your r is 0/1, but pi is probability. So individual cost is 1 if absolute error is greater than 0.5, otherwise 0. Then, this function calculates the average error rate. But remember, the cut-off has been set before you define your cost function.

Actually, I think it makes more sense if the choice of cut-off is determined by cost function.

$\endgroup$
0
$\begingroup$

The answer by @SLi already explains very well what the cost function you have defined does. However, I thought I would add that the cost function is used to calculate the delta value from cv.glm, which is a measurement of the cross validation error. However, critically delta is the weighted average of the error of each fold given by the cost. We see this by inspecting the relevant bit of the code:

for (i in seq_len(ms)) {
    j.out <- seq_len(n)[(s == i)]
    j.in <- seq_len(n)[(s != i)]
    Call$data <- data[j.in, , drop = FALSE]
    d.glm <- eval.parent(Call)
    p.alpha <- n.s[i]/n # create weighting for averaging later
    cost.i <- cost(glm.y[j.out], predict(d.glm, data[j.out, 
        , drop = FALSE], type = "response"))
    CV <- CV + p.alpha * cost.i # add previous error to running total
    cost.0 <- cost.0 - p.alpha * cost(glm.y, predict(d.glm, 
        data, type = "response"))
}

and the value returned by the function is:

  list(call = call, K = K, delta = as.numeric(c(CV, CV + cost.0)), 
    seed = seed)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.