Problem setting

Here is the problem. A customer is faced to several products corresponding to the needs he has expressed.

The goal is here to predict the probability of each product to be chosen.


The available data are the choices of previous customers having the same or different needs and presented and the products they were confronted to. The needs of the customers are defined by a vector of features $V_{needs}$. Each product is detailed by a vector of features $V_{product}$.

Proposed modelling

We proposed here to train a classifier which takes in input the vector of features of a given product, $V_{product}$, the vector of features of the client demands, $V_{needs}$, and a summary of the features of the other products presented to the customer. It outputs the estimated probability of the product to be chosen.

Namely, this classifier is essentially a Random Forest.

We have assessed this model via cross-validation with confusion tables, ROC and Precision-recall curves.

My question

Here, we do not want to classify a product as chosen or not chosen. We really want its probability to be chosen.

However, most of the metrics have been designed to assess its ability to classsify a points of a dataset as chosen or not.

So my question is: What are the metrics suited to validate the ability of the model to predict the right probability of a product ot be chosen (knowing all the products presented to the customer)?


One of the best metrics is the Brier Score. Here is it's Wiki description:

The Brier score is a proper score function that measures the accuracy of probabilistic predictions. It is applicable to tasks in which predictions must assign probabilities to a set of mutually exclusive discrete outcomes. The set of possible outcomes can be either binary or categorical in nature, and the probabilities assigned to this set of outcomes must sum to one (where each individual probability is in the range of 0 to 1).

And here's the link to the article:


| cite | improve this answer | |
  • $\begingroup$ This is an interesting idea. I will try it. Thanks $\endgroup$ – Pop Jan 22 '16 at 14:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.