# Metrics to asses the ability of a model to predict a probability

Problem setting

Here is the problem. A customer is faced to several products corresponding to the needs he has expressed.

The goal is here to predict the probability of each product to be chosen.

Data

The available data are the choices of previous customers having the same or different needs and presented and the products they were confronted to. The needs of the customers are defined by a vector of features $V_{needs}$. Each product is detailed by a vector of features $V_{product}$.

Proposed modelling

We proposed here to train a classifier which takes in input the vector of features of a given product, $V_{product}$, the vector of features of the client demands, $V_{needs}$, and a summary of the features of the other products presented to the customer. It outputs the estimated probability of the product to be chosen.

Namely, this classifier is essentially a Random Forest.

We have assessed this model via cross-validation with confusion tables, ROC and Precision-recall curves.

My question

Here, we do not want to classify a product as chosen or not chosen. We really want its probability to be chosen.

However, most of the metrics have been designed to assess its ability to classsify a points of a dataset as chosen or not.

So my question is: What are the metrics suited to validate the ability of the model to predict the right probability of a product ot be chosen (knowing all the products presented to the customer)?