3
$\begingroup$

What kind of data analysis technique / formula would you use to describe which is the fastest OS in this chart, Windows or Linux?

enter image description here

If Linux and Windows had equal speed on all tests, all bars would be 50% blue, 50% green.

The data is duration in milliseconds. Each number (e.g. Linux Startup) represents the mean of 1000 individual duration tests on that scenario. The results of each test individually is covered in separate charts.

This data represents an overview of the one set of data (Linux over the course of these 5 tests) and how it relates to the second set of data (Windows over the course of these 5 tests). The aim is to be able to say:

Over the 5 tests we Windows is faster. (You can see this in the chart because there is less green, but this is to prove it statistically).

Note:

The chart below shows the same data in real value terms, showing that one test dominates the others. While this is an important result, it does not weigh on the current question as the two results in the 'Refreshing Desktop' test still bear the same relation to each other as those in the other tests.

In the comments there has been some discussion on how to represent this data. The reason I represent it in stacked bars is because I want to show it without the overshadowing effect of the one anomalous data. Is there a preferred method to show this relationship?

enter image description here

Improve this question
$\endgroup$
21
  • 2
    $\begingroup$ This chart is meaningless to me because the vertical axis is unlabeled and I have no clue what, say, a 40% "like" would be. The only data analysis technique I know of that describes this graphic is chartjunk (and it's a super example of the species). $\endgroup$
    – whuber ♦
    Commented Sep 11, 2013 at 20:57
  • $\begingroup$ Ok, made a new chart, didnt think the last one through properly. $\endgroup$
    – Jack
    Commented Sep 11, 2013 at 21:07
  • 1
    $\begingroup$ I don't see why you would be using a stacked bar chart here. If these are times, why not have time on the Y axis & present the 2 bars side by side for each test. Setting that issue aside, Yes, you can average the 5 conditions, but there is a philosophical issue here as well: Are all situations equally important (ie, should you be using a weighted average, eg)? $\endgroup$
    – gung - Reinstate Monica
    Commented Sep 11, 2013 at 21:21
  • 3
    $\begingroup$ @gung That's more than a mere philosophical issue: it gets to the core reason why no statistical analysis can demonstrate the correctness of the desired conclusion. If one system is faster than the other on just a single test but otherwise is slower, it is still that case that people who highly value that one test would emphasize it to the complete neglect of all the others. This is an example of a multi-attribute valuation problem. $\endgroup$
    – whuber ♦
    Commented Sep 11, 2013 at 21:26
  • 3
    $\begingroup$ With the latest edit the chart has moved from the sublime to the ridiculous: the heights of the bars are meaningless (or else each is plotted on its own private axis, which is plain confusing). The chart should be replaced by the table of ten numbers (suitably rounded, one would hope) and left at that. It's just getting in the way of formulating an answerable question. $\endgroup$
    – whuber ♦
    Commented Sep 11, 2013 at 21:29

1 Answer 1

Reset to default
3
$\begingroup$

Displaying the results

You said in comments

I used stacking purely to display relationship of data. The actual quantity (duration) is irrelevant

Stacking makes the length comparison difficult; for the ones that are close together in time, you'd need to look at the numbers to see a difference. If the values don't matter, one option is to plot the logs of the ratios instead (but with axes representing the percentage difference)

With a plot more like this:

log-ratio plot

you get a better sense of the relative speed of the various tasks under the two systems.

However, it may be worth looking at plots where all of the individual values are represented (rather than just averages), so that an idea of the relative variability is also available.

Testing for a difference

The issue with any kind of statistical test is being very precise about what it is we mean to test.

In particular, what null and alternative hypothesis we're dealing with, and under what assumptions.

You have 5 numbers for each type of activity - are they just repetitions, or is there some kind of pairing across 5 different circumstances? Are you looking for a test within each kind of activity or something overall?

R code to (approximately) generate the above plot:

wl <- read.table(stdin(),header=TRUE)
Linux Windows Scenario
1.962 1.415 "Startup"
8.469 6.996 "Shutdown"
102.2 79.3 "RefreshDesktop"
1.777 2.866 "AccessInternet"
1.259 1.165 "WebcamStartup"

wl2 <- as.matrix(data.frame(wl[,1:2],row.names=wl[,3]))
wl3 <- wl2[,2]/wl2[,1]
opar <- par()
par(xaxt="n")
dotchart(log(wl3))
abline(v=0,col=8,lwd=2)
axis(side=1, at=log(c(1/1.6,1/1.4,1/1.2,1,1.2,1.4,1.6)),
    labels=c("60%","40%","20%","same","20%","40%","60%"),xaxt="s")
title(main="Relative speed of Windows and Linux",
   sub="             Linux slower by                        Windows slower by")
par(opar)
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.