What is the best metric for machine learning model to predict customer probability to buy I'm building a machine learning model to predict customer's propensity to buy (the likelihood that a customer buying a product). The purpose is rank the customers with probability score for customer targeting. Performance on binary outcome is not priority.
I'm seeking expert opinion what metric we are supposed to use in this context. (auroc, logloss, f1 ... etc?).  I have seen some conflicting opinions online.
What metric should I use if my dataset is highly unbalanced in this case? (buy vs not buy:  1:99).
Detailed explanations is highly appreciated!
 A: *

*Certainly, you shouldn’t use the common classification metrics like accuracy. They don’t do much good about having the probabilities correct.

*If you want to estimate the probabilities precisely, you need proper scoring rules (see other questions tagged as scoring-rules), like Brier score (squared error) or log loss (aka cross-entropy loss). There was recently an interesting paper by Hui and Belkin (2020) showing that using squared error as a loss function for a classier may give as good if not better results as compared to the “default” log loss.

*On another hand, you are saying that you want to use the probabilities to rank the customers, that’s a different problem. For ranking, you don’t care that much about the probabilities being correct, as far as they’re ordered correctly. There are specialised metrics like mean percentage ranking, mean reciprocal score, top-$k$ accuracy, precission@$k$, etc. Assuming it is a ranking problem, you probably should consider using specialized ranking algorithms as well.

The choice depends on how exactly you want to use the results and details about your data.
