2
$\begingroup$

I am trying to create a graph representing a trend in binary value distribution. I thought of my own solution but it somehow doesn't feel right. Here's what I'm trying to convey with my graph:

I have a list of items that are either correct or incorrect, and their order is random. I am comparing a couple of scoring algorithms, where higher score means more certainty that the item is correct. So when the list of items is sorted by score, items at the high-score end of the list should have higher correct frequency.

My graph: It would be a 2-axis graph, where x-axis would be increasing score values and y-axis would be correct/incorrect values. The graph itself would be a series of points and their trend line. How steeply the trend line rises would indicate how good is the algorithm at guessing correct items.

The graph feels a bit weird to me, especially the 2 value y-axis, but I can't think of anything better. Is the graph okay or is there a better way to represent the data?

$\endgroup$
  • $\begingroup$ How about putting time on the x-axis and aggregate score trends on the y-axis? $\endgroup$ – Mike Hunter Mar 27 '17 at 16:02
  • $\begingroup$ Time? I don't collect any time data, so I don't understand what you mean. $\endgroup$ – querti Mar 27 '17 at 16:09
  • $\begingroup$ Then how can you have a trend? Trends are defined by time, are they not? $\endgroup$ – Mike Hunter Mar 27 '17 at 16:10
  • $\begingroup$ Oh, I'm sorry, english is not my first language and I'm not very familiar with the terminology. What I mean is a linear trendline that you can create in excel and it can be created on basically any x-y graph $\endgroup$ – querti Mar 27 '17 at 16:13
  • 1
    $\begingroup$ Sounds like you might want to look at the concept of lift: (en.wikipedia.org/wiki/Lift_(data_mining)) and (stats.stackexchange.com/questions/17119/…). $\endgroup$ – Wayne Mar 27 '17 at 16:24
2
$\begingroup$

Sounds like you might want to look at the concept of lift. Here's another thread on Cross Validated that shows a lift chart in the accepted answer.

The Wikipedia entry first mentions a lift over random selection, but ultimately you'll probably want to look at a lift over the process that is currently used -- which probably isn't random, and therefore sets a higher bar.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Since you are dealing with only binary data, I would do Receiver Operating Characteristic curve (ROC) analysis (which by the way is related to lift analysis).

ROC is a chart of sensitivity (y-axis) vs 1-specificity (x-axis) and is quite common in the medical literature.

In your case, you would construct your ROC out of n classifiers and you would label each point by the name of the classifier (like the image). The closer you are to the upper left, the better your classifier is (see wikipedia image below). Since you have a bunch of random classifiers, I imagine many will be below the 1:1 line

ROC Space from Wikipedia

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.