2
$\begingroup$

I have a series of results of some hadoop experiments. There are more than 500 datapoints (each experiment) and I want to show the overall timing. I'm asking for an effective way to plot this dataset (1 graph per experiment obviously). I'm not sure about an 'ordered' scatter plot like this (actually there is no real order but the timing order, the X value here is just a progressive number): enter image description here

In this case I don't like the boxplot solution, sometimes the Q1 and mean are too close and the boxplot looks confusing. Maybe a normal distribution to visually show the mean and variance, there are tools for generate a normal distribution linegraph from data?

Other ideas?

$\endgroup$
  • $\begingroup$ Look for histograms, density plots or boxplots (specially if comparing two sets of results), one of these should do. $\endgroup$ – Firebug Oct 30 '16 at 0:53
1
$\begingroup$

You do not want a normal distribution if your data is in fact skewed, as it clearly is in your example. Here are some thoughts:

First let's get some data similar to yours

set.seed(1)
t <- 40+24*rexp(520)

so something similar to your chart comes from

plot(sort(t), ylim=c(0,250), ylab="Time (s)")

while you say you do not like something like

boxplot(t, ylab="Time (s)")

and a cumulative distribution would look like a reflection of your original chart

plot.ecdf(t, xlim=c(0,250), xlab="Time (s)")

so you might consider a histogram

hist(t, breaks=10*(0:25), xlab="Time (s)")

or perhaps a rather similar smoothed density, possibly with the mean of the data shown

plot(density(t), xlim=c(0,250), xlab="Time (s)", main="Density of times")
abline(v=mean(t), col="red")

with the last of these looking something like

enter image description here

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I'm sorry I don't get the Y value of the last chart, what's the density means? Are you talking about Kernel density estimation? Anyway, if 0.025 means 2,5%, it shouldn't be higher, since your histogram shows there are more than 150 points (out of 500) around 50 seconds? $\endgroup$ – alfredopacino Nov 1 '16 at 11:41
  • $\begingroup$ Yes, it is a kernel density estimate: see ?density for more details and how you can adjust it. If you integrate under the curve, you should get about $1$, as you do approximately with something like sum(density(t)$y[-1]*diff(density(t)$x)) $\endgroup$ – Henry Nov 1 '16 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.