1
$\begingroup$

Whilst I understand why median can be a better metric than mean in certain circumstances (e.g. you're creating/marketing a product, you want to use the median as it represents the biggest age group within your population and therefore real potential buyers, whereas mean age might be a number which doesn't correspond to anybody, or a very few people) I am wondering whether median is a good metric when it comes to measuring website performance.

Let's look at the below data representing page load time (in sec, the smaller the better):

enter image description here

Sample 1 and Sample 2 are identical, except for C4, meaning the website is only faster for 1 person. Yet, the median changes significantly between the 2 samples (nearly 40% improvement) whereas the mean is much more stable (less than 10% difference).

Accordingly my take on this is that mean is more accurate are representing the page load time.

Do you agree? / Is my observation biased because of the particular samples I have chosen or would mean be more representative than median no matter the samples?

$\endgroup$
  • $\begingroup$ Median age need not represent the biggest age group. In statistics "changes significantly" would be best established by a formal test; changes that are big, striking or notable can be described informally using any simple word. That aside, the median can be unstable too in small samples. First, it must be a sample value or the average of two values. More importantly, you are rounding to seconds, so the median may easily jump by 0.5 s. In a larger dataset I would expect the mean for data like yours to be a bit larger than the median, but why not use both? Why do you feel you have to choose? $\endgroup$ – Nick Cox Dec 18 '13 at 12:29
  • $\begingroup$ Thanks. Reason for wanting to choose one is that I like the idea of one number being more accurate than the other and that I get to use that number. Also, I'm gonna have to report this metric to management, and 2 different numbers with a whole explanation on statistics isn't going to cut it I'm afraid... What about the average value between the mean and median: does that make any sense? $\endgroup$ – Max Dec 18 '13 at 12:54
  • $\begingroup$ The average of mean and median is just what it says; it doesn't have a special name or much of a following. If people want to compromise between mean and median they usually go for a trimmed mean or some other resistant or robust summary; if your management can't handle mean and median that is way too esoteric for them. My main recommendation is that you calculate both mean and median, work out for yourself what works better for your data and purposes, and report that to management. Note that means are natural if totals are. $\endgroup$ – Nick Cox Dec 18 '13 at 13:00
2
$\begingroup$

You performed your analysis with a sample of only 10 data items. I recommend you try it again, but using a sample which contains a few 100 data items. Or try it again with another sample of 10 data items, but find a class of problem where more than just a single data item differs from one sample iteration to the next. What you will likely find (this is the usual reason why people often think that median makes a more reliable estimator than the mean) is that the outliers (i.e., the 8's and 9's in your sample) often vary wildly between samples.

Suppose you had one user who had a value of 25 for his website speed on one day, and a more "typical" value of 4 or 5 on some other day. That kind of thing happens all the time in real life. Think about your commute to work, as another example—most days it probably takes almost exactly the same amount of time every day, but every once in a while there is a giant traffic jam because of an accident on the road, or the train is delayed for a large and random amount of time because another train ahead of you experienced a mechanical problem or someone accidentally fell onto the tracks a few stops in front of you, and the whole system grinds to a halt during rush hour while the authorities deal with the problem.

Outliers in distributions like these, which represent a process that has gone "wrong" in some unpredictable way on one particular occasion, tend to be large and variable in size and frequency, and they usually have a much larger effect on the calculation of the mean than the median. Thus, although you've detected a counterexample that works in this particular case, in general, the median usually offers a more robust estimator (in the sense of being more stable from one sample to the next) than the mean.

$\endgroup$
  • $\begingroup$ Say, rather, a sample of 10 or 100, not 10 or 100 samples. Both usages are common across science, but that's statistics speak. (Trivium: I walk to work and the time hardly varies....) $\endgroup$ – Nick Cox Dec 18 '13 at 13:06
  • $\begingroup$ @Nick Cox: I agree. I've edited my response to be more clear. (And you're right, my commuter example doesn't work quite as well for walkers, bikers, or people on roller skates, etc.) $\endgroup$ – stachyra Dec 18 '13 at 13:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.