I have an unbalanced classification problem (95% "0", 5% "1") regarding quality control."0" means "no problem" and "1" means "problem".

I'm not predicting real cases one by one, this is, my client provides me with a new list of, let's say, 10000 new instances , and he expects from me a list of 100 cases, with as many "1" as possible, because he has no time to fully inspect 10000 instances.

I have two questions:

a) What optimization metric will you use? I've been using AUPRC, AUROC, and even Logloss, with very similar results. Theoretically, wich one will be more appropiate?

b) Once I send the list of 100, my client returns me two lists: the one I send with the real result "0" and "1" and another of aprox the same size but randomly selected, also with the result of the inspection. How would you update the training data? Random instances only or all the instances, the 100 I send plus the 100 random?

As of today, n=20000, p=50, lots of NA, mostly cat features. I'm using catboost.


2 Answers 2


a) At least from your perspective, you don't really care about overall performance measures. Both AUROC and AUPRC are overall measures. (AUPRC is simply average precision; see Menon and Williamson (2016), Lemma 52.) You care most about the precision (fraction of true-problem cases) in the top 1% (100 out of 10,000) of cases predicted to be problems, when the overall prevalence of problems is 5%. You thus might want to consider a customized loss function that recognizes this peculiarity of your situation.*

As you had problems with customized loss functions that focused on the top 1% of cases, you might consider using a flexible proper scoring rule that puts emphasis on that end of the rankings while still evaluating all cases. Merkle and Steyvers describe a beta family of proper scoring rules with two parameters $\alpha$ and $\beta$. The case $(\alpha = 0, \beta = 0)$ is logarithmic scoring, and $(\alpha = 1, \beta = 1)$ is the Brier score. Choosing values of $\alpha$ and $\beta$ that differ from each other gives differential weighting of high versus low probability predictions while maintaining the advantages of proper scoring. There is an R package scoring that implements this family of scoring rules, among others. Section 9 of Menon and Williamson (2016) provides a broad overview of this problem of "ranking the best."

b) The updating issue might require further knowledge about the subject matter at hand. Your client presumably is altering protocols to reduce the number of problems, so it's not clear whether relations of features to outcome will be consistent over time. That possibility would need to be taken into account. Note that what you are getting from the client is something similar to case-control analysis in clinical studies, although I haven't thought through carefully how that observation would inform how you might proceed. In general there isn't much to be gained by throwing away data, so I don't see why you shouln't update with all instances returned to you by your client (provided that the possibility of changes of the model over time is considered).

*How wise this is from your client's perspective is another question. Even if all 100 of your selections are true problems, there is still the issue of the remaining 4% of true problems that aren't being examined.

  • $\begingroup$ In fact, I did a customized loss function, but i was getting worse results. I think that the problem was in the size. As n~20000, and I'm doing hyperparameter optimization crossvalidating with 5 folds, each fold is ~4000, and the 1% is 40 cases only, so the metrics has high sensibility because this short number of cases. My client sometimes send lists with 1000 instances,1% is 10 , and sometimes with 100.000,1% is 1000 instances. $\endgroup$ Apr 15, 2019 at 12:43
  • $\begingroup$ @PeterTschuschke I modified my answer in response to your clarification. $\endgroup$
    – EdM
    Apr 15, 2019 at 18:38

a) In the case of unbalanced data, AUPRC should be the best choice as a metric, since in this case AUROC can be easily inflated to values close to 1.

b) Sounds interesting! You should definitely include the cases where your classification decided wrongly, since you can "learn more" out of these if you include them in your training set. You should take a read into reinforcement learning, since your problem is a mixture of supervised learning with prediction on new data and reinforcement learning afterwards.

  • $\begingroup$ AUROC is a common choice for unbalanced data exactly because it is agnostic to the class priors and evaluates the general discriminative power of a model. (See this article by Fawcett) $\endgroup$
    – Scholar
    Apr 15, 2019 at 10:09
  • $\begingroup$ That's exactly the point: If you know that your data is imbalanced, you should use AUPR, since AUROC is agnostic to class imbalance and precision on the other hand is much more sensitive to false positives. AUROC alone couldn't tell you if your classifier is a good choice. $\endgroup$
    – Edgar
    Apr 15, 2019 at 13:39
  • $\begingroup$ I think you are missing the point, probably because you treat this question as if it has a definitive and non-subjective answer. As I mentioned, the AUROC evaluates the overall discriminative power of a model and represents the probability of ranking a case instance above a control. This is a very informative metric, irrespective of the class imbalance. $\endgroup$
    – Scholar
    Apr 15, 2019 at 14:07
  • $\begingroup$ hm. So if I know my classes are imbalanced and I want to optimize the choice of my classifier, shouldn't I take precision into account? Don't I miss something if I look at AUROC only? On the other hand, what do I gain if I take AUROC in addition to AUPR? $\endgroup$
    – Edgar
    Apr 15, 2019 at 14:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.