Which metric to use in an ordering problem? auPR / ROC / Lift? I need to order Users from most likely to perform a binary action X in the next n days, to the least likely.
To solve the problem I'm training a XGBoost classifier and ordering the Users by the predicted probability.
The probablity itself doesn't matter, only that the order is correct. 
What would be a good method for evaluating this model?  
I thought about splitting the data to percentiles and checking the lift in each percentile when I expect to see a higher lift so the percentile increases.
Problems with this approach - No "one number" metric, can't evaluate across different datasets.  
Thought about using auPR but I'm not sure what exactly the auPR number means.
I read here that its ...The probability that if a “positive” edge is selected from the ranked list of the method, then an edge above it on the list will be “positive” 


*

*Can someone explain why this actually means what it says?

*Is this the most fit metric to optimize in my case?

*Other suggestions?
 A: This appears to be a ranking problem. Therefore it would be more relevant to look at metrics like the (normalised) Discounted Cumulative Gain, Precision@K and other information retrieval performance metrics. Ranking is a very different beast compared to both regression and classification; I would suggest one to first read upon information retrieval a bit to built some intuition. In the modelling task described, users are the documents to be retrieved. I have found reading a bit upon Bradley-Terry models helpful as they address a somewhat similar problem but taking an approach that is conceptually closer to a "standard probabilistic classification" modelling task. Another thing that helped things "click" for me was the difference between list-wise and pair-wise metrics; the paper by Li on A Short Introduction to Learning to Rank was a very good intro on the matter.
You mention XGBoost; XGBoost natively supports rank-related learning objectives as well as evaluation metrics like the NDCG and other rank-related indicators so it should play nicely to the requirements of this task. The XGBoost GitHub repo has a ranking tutorial that should help you getting started.
