Bias-Variance tradeoff for classifying unbalanced classes I would like to use Bias-Variance trade-off to evaluate training set size in a classification problem. There are two classes which are not balanced (~70/30) and it seems that the common use of misclassification error is not good enough.
Which performance measures should I use in this case? 
 A: You could use precision or recall measures, or F1 which is a combination of the two.
Precision is the ratio of true positives, divided by the number of predicted positives (= the sum of the true positives and the false positives).
Recall is the ratio of true positives, divided by the number of actual positives (= the sum of the true positives and the false negatives).
The values for precision and recall you want depend on your problem. For example, if you only want to predict y = 1 when you are very confident, use a higher precision (and lower recall).
If you want a single number evaluation, the F1 score is calculated as follows: 2 * ((P*R)/(P+R)) with P being precision and R being recall.
A: I'm a bit confused about why you're mentioning Bias-Variance tradeoff, but F1 score is indeed a good simple metric to avoid the unbalanced problem. Suppose you label your classes as positive and negative and their respective distribution is 30 / 70. If your classifier always predicts the negative class, then here are the true positive, true negative, false negative, true negative values:
TP = 0
TN = 70
FN = 30
FP = 0
Thus, the classifier's accuracy will be:
ACC = (TP + TN) / (TP + TN + FP + FN) = 70%
But the F1 score will be F1 = 2*TP / (2*TP + FP + FN) = 0, which clearly tells that this "dumb" classifier cannot predict the the positive data.
