How useful is estimate of accuracy for cross-validation in case of imbalance in class distribution I have about 4000 instances of one class and 38000 instances of another. I used the DAAG library and I got the following result:
> fit2 <- glm(formula=y~x1+x2,data=rec,family=binomial)
> library(DAAG)
> cv.binary(fit2)

Fold:  5 3 9 10 1 7 2 4 8 6
Internal estimate of accuracy = 0.911
Cross-validation estimate of accuracy = 0.911

In case of class imbalance in general, accuracy is not a very useful measure. Is this the same accuracy? How useful is the result I obtained?
 A: You are correct that overall accuracy is not a useful measure in this case, since simply predicting the more common class all the time will give an accuracy of 38000/42000=0.905. There are several approaches you can use in this case:


*

*Measure the accuracy separately for each class, and take the mean.

*Use the Matthews Correlation Coefficient, which measures the correlation between your predictions and the real class labels.

*Change your cross-validation procedure so that the same number of each class is held out for testing on each fold. This may require some recalibration of the learned model, since the model will be expecting unbalanced data (and so may have a strong prior about which class to predict).

*Use more data about the problem domain to adjust your cost function. Are mistakes in either class equally bad? Rather than fitting a model which counts all mistakes equally, your model could weight mistakes in the smaller category higher than mistakes in the larger category.


I'm not very familiar with R and DAAG, so I can't offer specifics about how these would be implemented in this framework.
