I am trying to fit a logistic regression model in R to classify a dependent variable as either 0 or 1. I have a dataset of around 2000 observations and decided to split it in half (training and testing).

After deciding which variables to include in my model, I subsetted the data and fitted the logistic regression as follows:

clf <- glm(y~.,data=df,family='binomial')

Then, I tested the classifier on the testing set (1000 observations) and got 0.75 accuracy score.

results <- ifelse(predict(model,testdf,type='response') > 0.5,1,0)
error <- mean(r_results != results)
print(1-error) #prints out 0.74984

After this step, I decided to crossvalidate using the boot package


# K-fold CV
error_cv = NULL

# Cost function for binary variable (as suggested by the R documentation)
cost <- function(r, pi = 0) mean(abs(r-pi) > 0.5)

for(i in 1:10)
    error_cv[i] <- cv.glm(df,clf,cost,K=10)$delta[1]


Now, here is where I encounter a problem. K-fold cross validation as I understand it, does the following (quote from Wikipedia):

"In k-fold cross-validation, the original sample is randomly partitioned into k equal sized subsamples. Of the k subsamples, a single subsample is retained as the validation data for testing the model, and the remaining k − 1 subsamples are used as training data. The cross-validation process is then repeated k times (the folds), with each of the k subsamples used exactly once as the validation data."

Why is it that cv.glm() takes as an argument my already-fitted model? I don't understand what it is doing. Furthermore, if the data argument is equal to the training set, I get error rates of around 0.2, whereas if I set data = testdf I get error rates of around 0.4. Since the two sets, df and testdf, have been split randomly, I cannot explain this large difference and I cannot explain why cv.glm() does not (apparently) do the fit and test process it is supposed to do.

What am I missing?

  • $\begingroup$ Interesting question, nicely asked ... I don't get it either ... $\endgroup$ – citraL Sep 9 '15 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.