Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite

N.B., The questions require a bit of setup, which I've tried to present abstractly; please feel free to comment on anything that needs clarification.

I have problem where I am predicting the value of several actions for an ensemble of systems that satisfy a particular set of constraints, then observing the each of action values (via simulation) for sample instances of that system.

I'm presenting the results as a series of boxplots where the observed distributions are vertical boxes spreading in the y-direction, and are located horizontally by their predicted values. Each plot in the series compares a different prediction method.

Question 1: I think this provides a pretty good visual argument about the relative worth of the prediction methods, but it's not exactly quantitative. I need a statistic that compares the different methods ability to accurately order the outcomes, including accounting for "close" predictions mis-orders when outcomes are also close being not as "bad" as "not close" predictions mis-orders. Suggestions?

Question 2: I also have data for a large sample of different constraint sets - i.e., many different ensembles; in this case, each system has a different ordering for actions (not just by prediction method). Any recommendations for visualizing the results comparing the different prediction methods?

share|improve this question
I am trying the Spearman rank correlation currently. This still doesn't account for the "scale" part of my question - namely, that estimating "small" differences and misordering should be better than estimating "large" differences and misordering. – Carl Oct 25 '11 at 14:32
Please add this as a part of your question and remove the answer. – mpiktas Oct 26 '11 at 10:44
Maybe these lines of mine help to further clarify your question (which I'm pretty sure I won't be able to answer then :) but hopefully it helps some others to do so...) One point where I get stuck while reading your question is this: "the value of several actions". Do you assign a value to an action? Is this based on reaching some goal and you give marks to the actions like "helpful", "not helpful", "neutral"? – psj Oct 27 '11 at 13:39
@psj: to clarify a bit: in my problem, there are several options to pursue. The exact outcome of pursuing any option is unknown, but it's all on the same quantitative scale - e.g., option A might yield 5.4 and option B might yield 3.1 - and that order of options on that scale indicates which one is best, second best, etc down to worst. I have several methods which predict scores on some other scale - that is, I'm not predicting the score or something that can be mapped to it readily - but the order on that prediction scale reflects the order on the actual scale. – Carl Nov 5 '11 at 13:13
@mpiktas: that's not the way I understand this to work. I figured out that answer, incomplete though it is to my requirements, after I formulated the question. "Use the spearman rank to compare ordering performance", based on what I've learned elsewhere on my own, is a valid answer to my question. – Carl Nov 5 '11 at 13:17

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.