AUPRC vs AUROC and updating training set in quasi-classification problem I have an unbalanced classification problem (95% "0", 5% "1") regarding quality control."0" means "no problem" and "1" means "problem".
I'm not predicting real cases one by one, this is, my client provides me with a new list of, let's say, 10000 new instances , and he expects from me a list of 100 cases, with as many "1" as possible, because he has no time to fully inspect 10000 instances. 
I have two questions: 
a) What optimization metric will you use? I've been using AUPRC, AUROC, and even Logloss, with very similar results. Theoretically, wich one will be more appropiate?
b) Once I send the list of 100, my client returns me two lists: the one I send with the real result "0" and "1" and another of aprox the same size but randomly selected, also with the result of the inspection. How would you update the training data? Random instances only or all the instances, the 100 I send plus the 100 random?
As of today, n=20000, p=50, lots of NA, mostly cat features. I'm using catboost. 
 A: a) At least from your perspective, you don't really care about overall performance measures. Both AUROC and AUPRC are overall measures. (AUPRC is simply average precision; see Menon and Williamson (2016), Lemma 52.) You care most about the precision (fraction of true-problem cases) in the top 1% (100 out of 10,000) of cases predicted to be problems, when the overall prevalence of problems is 5%. You thus might want to consider a customized loss function that recognizes this peculiarity of your situation.*
As you had problems with customized loss functions that focused on the top 1% of cases, you might consider using a flexible proper scoring rule that puts emphasis on that end of the rankings while still evaluating all cases. Merkle and Steyvers describe a beta family of proper scoring rules with two parameters $\alpha$ and $\beta$. The case $(\alpha = 0, \beta = 0)$ is logarithmic scoring, and $(\alpha = 1, \beta = 1)$ is the Brier score. Choosing values of $\alpha$ and $\beta$ that differ from each other gives differential weighting of high versus low probability predictions while maintaining the advantages of proper scoring. There is an R package scoring that implements this family of scoring rules, among others. Section 9 of Menon and Williamson (2016) provides a broad overview of this problem of "ranking the best."
b) The updating issue might require further knowledge about the subject matter at hand. Your client presumably is altering protocols to reduce the number of problems, so it's not clear whether relations of features to outcome will be consistent over time. That possibility would need to be taken into account. Note that what you are getting from the client is something similar to case-control analysis in clinical studies, although I haven't thought through carefully how that observation would inform how you might proceed. In general there isn't much to be gained by throwing away data, so I don't see why you shouln't update with all instances returned to you by your client (provided that the possibility of changes of the model over time is considered).

*How wise this is from your client's perspective is another question. Even if all 100 of your selections are true problems, there is still the issue of the remaining 4% of true problems that aren't being examined. 
A: a) In the case of unbalanced data, AUPRC should be the best choice as a metric, since in this case AUROC can be easily inflated to values close to 1.
b) Sounds interesting! You should definitely include the cases where your classification decided wrongly, since you can "learn more" out of these if you include them in your training set. You should take a read into reinforcement learning, since your problem is a mixture of supervised learning with prediction on new data and reinforcement learning afterwards.
