1
$\begingroup$

I'm not a stat person, so I apologize if this question is trivial.

I have data obtained from several simulation methods that need to be compared to experimental data (which presumably represent the "correct" values). Something like this:

Sim method 1 Sim method 2 ... Experiment
result result ... result
result result ... result
.... .... ... ....

Note that the individual numerical values for each method don't really matter -- we only care about the difference with experimental values (i.e. we're trying to determine which simulation method is closer to experiment, on average).

For something like this, people simply compare RMSDs (root mean square deviations) or some other similar average of the differences. However, because I'm comparing a lot of methods (and for a lot of categories) just including a table of RMSDs would kind large and unwieldy, so I was looking for a more visually appealing way of conveying the same information.

I thought box-plots (or something similar) might work well in this case, with the box centered on some sort of "average" deviation from exp. values, and error bars depecting the variance of the data points from that avarge. However, I'm not sure what's the best way to compute the average and variance here, and whether the average should be something like a simple mean or an RMSD, whether the variance should be symmetrical or assymetrical etc.

I'm looking for recommendations. What is the best way to visualize this sort of data?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Are the data matched such that a given run will yield the output for each sim method & the experimental result, or is it just that you have N sim 1 results, & N unrelated sim 2 results... etc? $\endgroup$
    – gung - Reinstate Monica
    Commented Jul 7, 2021 at 18:55
  • $\begingroup$ If I understand your question correctly, yes, they all compute the same properties. I.e. N physical properties are computed with sim 1, the same N physical properties are computed with sim 2, etc... and then the same N physical properties are measured experimentally. For the purposes of this study, experimental measurements are considered "exact." $\endgroup$
    – johnymm
    Commented Jul 7, 2021 at 23:48
  • 1
    $\begingroup$ What I mean is something like this: imagine comparing the settings for a camera. I pick out a bunch of settings & take a picture of a flower, then I use the next group of settings & take a picture of the same flower, etc. for k groups of settings. Then I repeat the whole process, taking a picture of a chair, then I do it again, taking a picture of toy, etc. I can compare how sharp the resulting 1st pictures are between each of the settings, & compare the 2nd pictures from each of the settings, etc. because in each case, they are pictures of the same thing. $\endgroup$
    – gung - Reinstate Monica
    Commented Jul 8, 2021 at 1:02
  • $\begingroup$ Yes, the analogy works, more or less. If settings are the simulation methods, they are all comparing the same thing. I have k number of camera settings, and n number of objects that I take a picture of using each setting. Then, for example, I want to determine how sharp each setting is by extracting some "sharpness value" x from each picture -- and comparing to some "ideal" value of x. Each object has a different value of x, though ideally they should be the similar across settings, unless a certain setting is doing something fundamentally wrong. I hope that clarifies it. $\endgroup$
    – johnymm
    Commented Jul 8, 2021 at 2:51

1 Answer 1

Reset to default
1
$\begingroup$

Unless n is small (say, <15), I think using boxplots is perfectly reasonable here. One suggestion I would make, however, is not to make a boxplot for the experimental readings. I gather you think those values are correct by definition. You want to compare the simulation readings to those. Moreover, you have dependent (within-unit) data. Thus, I would calculate the difference between each reading from a given run (photographed object) and the corresponding reading from the experiment, for each simulation and plot those. You will end up with a boxplot of differences for each simulation, and no boxplot for the experiment. You can see an example of what I have in mind in my answer here: Is using error bars for means in a within-subjects study wrong? (although that figure uses barcharts instead of boxplots).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.