I have two ranking lists(A & B) and a list C combined from the two lists, I want to use ROC curve to get the contribution of A and B to build C. How does that work?
Example, For list A the order is:
2,6,3,4,1,9,5,7,10,8
list B is:
3,1,4,2,5,8,6,7,9,10
list C is:
1,2,3,4,5,6,7,8,9,10
How can I draw two ROC curve to represent the contribution of list A and B to list C.
I assume that you want to evaluate the performance of a model, i.e. given many cases where one has some hints what the correct order is (or maybe even the full correct order) and two models MA and MB, then we let MA and MB run on all th cases and compare how 'wrong' their orders were in comparison to the correct order (or at least a part of it).
The important question is: Is it true that only a few first items are important to guess correctly, i.e. is the true order something like
1,2,3 and the rest is not so important
or is it mandatory that the predictor really guesses the whole order
1,2,3,4,5,6,7,8,9,10
correctly?
First case: only a few items in front are important.
Then one can actually do something with a ROC curve. First of all I will elaborate on a real world example where I have done so and secondly, I will describe how to use a ROC-like technique in order to evaluate how good predictors MA and MB are.
'Real world' example:
Let us assume that we are Amazon. Assume that we have only 4 items X,Y,Z,W that we sell. We want to sell as much stuff as possible. We have observed customers visiting our website. Let us assume that the customers were logged in already and browsed our offer. Since the customer was logged in, at that point in time, we had a solid knowledge about its past behaviour (i.e. which items does he/she like? which does he/she not like? age, gender, time of the day, ...). Using all that information, two different very smart data scientist come up with two different models and each one claims that his/her model is better than the other. How do we decide which model we take to production?
We assume that the models predict a 'rank' of the items and we assume that if an item gets rank 1, then it is 'very present' on the website, i.e. it gets advertised the most and if it gets rank 4, then it is advertised only very little, i.e. all itms fight for rank 1 (the most important one) and if that is spent then they fight for rank 2 and so on.
We decide to restrict ourselves only to the case where the customer finally bought an article (for the other case one also has to come up with a metric but you will get the idea along the way). So let us say that the customer bought items X and Z. So for the models MA and MB it would be desirable that items X and Z have a low rank so that these are presented very vividly to the customer while they should place items Y and W on high ranks in the 'back' of the order.
A second assumption (since you speak about ROC) is that the items are not just ranked in some weird way by the models MA and MB but rather those models spit out (for every item) a 'trust' or probability value between 0 and 1 that the customer will buy the article they recommend and the order is then of course to place the item with the highest value on the first rank, the next item is the one with the second highest trust value and so on. I.e. the situation could be like this:
ITEM | MODEL_A | MODEL_B
X | 0.8 | 0.1
Y | 0.2 | 0.2
Z | 0.9 | 0.3
W | 0.1 | 0.4
The numbers do not necessarily have to add up to one or so.
Now we pretend (just for the evaluation, this is not happening in the real world!) that we only have limited space available on the website. Lets say we believe model A for a second. How do we decide which items to put on the website and which to leave out? Of course, we would go for items X and Z (and the model would be correct because the customer bought these items) but how do we decide that actually? We take some threshold value (say 0.6) and show all items on the webpage that have a trust value higher than the threshold. I.e. for model B we would not show any item on the website at all. Now yo ucompute the ROC curve as follows:
for every threshold t you go through all the cases and see what percentage of items that all customers bought (i.e. were interested in) on the webpage
Then you draw the curve and the one that 'bows up' more is the better model.
Second case: the complete order is important.
In this case you need to define a metric on 'how far off' a model is given the true, correct, desired order. In that case I would start with something simple like:
true order = 1,2,3,4,5,6,7,8,9,10 model order = 2,6,3,4,1,9,5,7,10,8
then the first 2 is off by 1 the 6 is off by 4 the 3 is off by 0 ...
i.e. the average 'off' is (1 + 4 + 0 + ...)/10.
Then of course we need more information on how you will use that. For example: the order
2,3,4,5,6,7...
is off by 1 but it is really close to the desired order 1,2,3,4,... is that to punish much or little? You have to ask yourself a few of these questions and then tweak the metric until it finally characterizes the distances of the orders that you want.
