I have a problem of binary classification with class imbalance for the positive label (about 10:1 odds), except that in doing the classification I want to rank the data by confidence of prediction.

In this classification I want the classifier to prefer recall over precision so that it's more aggressive at finding the rare positive examples, so the natural method to combine these is the $F1_\beta$ where say $\beta = 2$.

My question is whether of not the following line of reasoning for formulating an analogous metric for real-valued predictions makes sense, and if it doesn't where and why:

1) To get confidence ratings, formulate the task as regression instead of classification, using say, logistic regression or random forest regression to make predictions. The regressor will learn from examples with responses that are either 1's or 0's, and so predictions that are closer to 1's or 0's are more confident.

2) Now that it's regression, $F1$ doesn't apply. Since I care more about the model being correct with the rare positive examples, I want to give errors for positive examples more weight. Here the error is RMSE.

I suppose I could just use a weighted average of the errors, but alternatively I could keep two different error rates, call them $e_+$ and $e_-$, one for errors on positives and one for errors on negatives. Taking $1-e_+$ and $1-e_-$ gives me an estimate of accuracy for each class.

3) Given that I have two rates, where I want to give one rate more preference, the $\beta$-weighted harmonic mean ( identical in formula to the $F1$ score) seems appropriate.

Does this metric make sense?

Mathematically it would be:

Given $n$ labeled data samples $\mathbf{X} = \{(X_1, l_1),...,(X_n,l_n)\}$ with sets of indices $L_0 = \{i \| l_i = 0\}$ and $L_1 = \{i \| l_i = 1\}$

Learn some real-valued model $M:\mathbb{R}^D \rightarrow [0,1]$ where $D =|X_i|$ $\forall i$

Calculate RMSE error rates $e_0$ and $e_1$ where $e_0 = \sqrt{\frac{\Sigma_{i \in L_0}(M(X_i)-l_i)^2}{\|L_0\|}}$ and $e_1 = \sqrt{\frac{\Sigma_{i \in L_1}(M(X_i)-l_i)^2}{\|L_1\|}}$

Calculate accuracies $a_0 = 1 - e_0$ and $a_1 = 1 - e_1$

For given $\beta$ calculate:

$$Score(M, \mathbf{X}) = (1+\beta^2)\frac{a_0a_1}{\beta^2a_0 + a_1}$$


There are various performance measures. Although F1-scores are claimed to account for class imbalance, these scores are unreliable under certain circumstances, e.g. class imbalance.

Fawcett et al. (2005) suggested and investigated the use of AUC-values from ROC-curves in testing classifier performance (introductory article). For binary classification using a support vector machine, I implemented these measures along with accuracy for MATLAB found here.

Now, for your considerations about the performance measures. In theory your outlined equations look fine to me. However, I am uncertain if they hold in practice.

When using regression, RMSE is perfectly fine. If you want to be able to have further measures, a suggestion from my side is to transform your continuous labels into discrete labels. By that you are able to classify the data and labels. If applicable, the easiest is to apply some thresholding, e.g. the mean of targets and non-targets or percentiles.

  • $\begingroup$ Thanks! The Fawcett paper was a great reference. Using AUC on thresholded classification seems like metric I was looking for. Especially since ROC/AUC are invariant to class-imbalance. $\endgroup$ – Taaam Nov 13 '15 at 17:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.