N.B., The questions require a bit of setup, which I've tried to present abstractly; please feel free to comment on anything that needs clarification.
I have problem where I am predicting the value of several actions for an ensemble of systems that satisfy a particular set of constraints, then observing the each of action values (via simulation) for sample instances of that system.
I'm presenting the results as a series of boxplots where the observed distributions are vertical boxes spreading in the y-direction, and are located horizontally by their predicted values. Each plot in the series compares a different prediction method.
Question 1: I think this provides a pretty good visual argument about the relative worth of the prediction methods, but it's not exactly quantitative. I need a statistic that compares the different methods ability to accurately order the outcomes, including accounting for "close" predictions mis-orders when outcomes are also close being not as "bad" as "not close" predictions mis-orders. Suggestions?
Question 2: I also have data for a large sample of different constraint sets - i.e., many different ensembles; in this case, each system has a different ordering for actions (not just by prediction method). Any recommendations for visualizing the results comparing the different prediction methods?