15
$\begingroup$

Suppose I have samplings from two distinct populations. If I measure how long it takes each member to do a task, I can easily estimate the mean and variance of each population.

If I now hypothesise a random pairing with one individual from each population, can I estimate the probability that the first is faster than the second?

I do have a concrete example in mind: the measurements are timings for me cycling from A to B and the populations represent different routes I could take; I'm trying to work out what the probability is that picking route A for my next cycle will be faster than picking route B. When I actually do the cycle, I've got another data point for my sample set :).

I'm aware that this is a horribly simplistic way to try to work this out, not least because on any given day the wind is more likely to affect my time than anything else, so please let me know if you think I'm asking the wrong question...

Improve this question
$\endgroup$
6
  • $\begingroup$ This is can be done via simple binomial testing & @Macro has a good answer. However, one issue is with the samples themselves: is there anything that could affect your decision to take route A or route B? In particular, do you like to take route A when the roads are dry, the wind is at your back, and dinner's waiting? :) Just be careful of anything that could affect outliers in either set or that could bias the samples in some way. For instance, try setting up your sampling plan ahead of time, with consideration for any need to vary (e.g. safety). $\endgroup$
    – Iterator
    Aug 24, 2011 at 20:57
  • $\begingroup$ One other consideration: Suppose that you have two routes with very similar means and neither dominates the other in terms of the probability that it will be faster. E.g. one is always either 10 or 20 minutes, while the other is always precisely 15 minutes. You may find it better to penalize greater uncertainty (e.g. standard deviation), or to favor one that is more likely to take less than some threshold of time. Your question as-is is fine; I am merely suggesting a future refinement. $\endgroup$
    – Iterator
    Aug 25, 2011 at 4:32
  • $\begingroup$ The statistical question is fine, but if you want to work out the probability which route is faster, I should suggest measuring the lengths of the routes. If the terrain is not hilly then the shorter route will always be faster. $\endgroup$
    – mpiktas
    Aug 25, 2011 at 11:59
  • $\begingroup$ If wind is an important factor, and if the wind speeds are related for the two routes, then it would seem that a person would need information about the dependence between A and B to answer the question accurately. You would need bivariate data for that, and it's hard to ride two paths at the same time. You could enlist someone else to help you collect data, but then would need to account for the variability between riders. In the case A and B are independent, the answers below are great. $\endgroup$
    – user1108
    Aug 25, 2011 at 16:02
  • $\begingroup$ Put another way: if I'm trying to decide which path to take, one's through a tunnel, one's through a field, and the wind's blowing like crazy, I may very well choose the field even if it does horribly worse on average. $\endgroup$
    – user1108
    Aug 25, 2011 at 16:16

3 Answers 3

Reset to default
13
$\begingroup$

Solution

Let the two means be $\mu_x$ and $\mu_y$ and their standard deviations be $\sigma_x$ and $\sigma_y$, respectively. The difference in timings between two rides ($Y-X$) therefore has mean $\mu_y - \mu_x$ and standard deviation $\sqrt{\sigma_x^2 + \sigma_y^2}$. The standardized difference ("z score") is

$$z = \frac{\mu_y - \mu_x}{\sqrt{\sigma_x^2 + \sigma_y^2}}.$$

Unless your ride times have strange distributions, the chance that ride $Y$ takes longer than ride $X$ is approximately the Normal cumulative distribution, $\Phi$, evaluated at $z$.

Computation

You can work this probability out on one of your rides because you already have estimates of $\mu_x$ etc. :-). For this purpose it's easy to memorize a few key values of $\Phi$: $\Phi(0) = .5 = 1/2$, $\Phi(-1) \approx 0.16 \approx 1/6$, $\Phi(-2) \approx 0.022 \approx 1/40$, and $\Phi(-3) \approx 0.0013 \approx 1/750$. (The approximation may be poor for $|z|$ much larger than $2$, but knowing $\Phi(-3)$ helps with the interpolation.) In conjunction with $\Phi(z) = 1 - \Phi(-z)$ and a bit of interpolation, you can quickly estimate the probability to one significant figure, which is more than precise enough given the nature of the problem and the data.

Example

Suppose route $X$ takes 30 minutes with a standard deviation of 6 minutes and route $Y$ takes 36 minutes with a standard deviation of 8 minutes. With enough data covering a wide range of conditions, the histograms of your data might eventually approximate these:

Two histograms

(These are probability density functions for Gamma(25, 30/25) and Gamma (20, 36/20) variables. Observe that they are decidedly skewed to the right, as one would expect for ride times.)

Then

$$\mu_x = 30, \quad \mu_y = 36, \quad \sigma_x = 6, \quad \sigma_y = 8.$$

Whence

$$z = \frac{36 - 30}{\sqrt{6^2 + 8^2}} = 0.6.$$

We have

$$\Phi(0) = 0.5; \quad \Phi(1) = 1 - \Phi(-1) \approx 1 - 0.16 = 0.84.$$

We therefore estimate the answer is 0.6 of the way between 0.5 and 0.84: 0.5 + 0.6*(0.84 - 0.5) = approximately 0.70. (The correct but overly precise value for the Normal distribution is 0.73.)

There's about a 70% chance that route $Y$ will take longer than route $X$. Doing this calculation in your head will take your mind off the next hill. :-)

(The correct probability for the histograms shown is 72%, even though neither is Normal: this illustrates the scope and utility of the Normal approximation for the difference in trip times.)

Improve this answer
$\endgroup$
19
  • $\begingroup$ if you have iid realizations from each distribution then what is the advantage of using the normal approximation rather than a monte carlo re-sampling approach (my answer) to estimating $P(X>Y)$? $\endgroup$
    – Macro
    Aug 24, 2011 at 23:18
  • $\begingroup$ @Macro: if the data can be reduced to summary statistics for the Q of interest, one can store less data... just a thought. $\endgroup$
    – Iterator
    Aug 24, 2011 at 23:25
  • $\begingroup$ Sorry, my brain was fried by heat & I missed the obvious answer. You are each answering different questions. The bootstrap method you gave estimates $P(X > Y)$, while @whuber is considering the difference in the mean times, which isn't the same. It isn't too hard to construct a case where option $Y$ is shorter than option $X$ 60% of the time, but the mean for $Y$ is greater than the mean for $X$. $\endgroup$
    – Iterator
    Aug 24, 2011 at 23:53
  • $\begingroup$ FWIW: @whuber is describing Student's t-test for the difference in means between two samples with different standard deviations. $\endgroup$
    – Iterator
    Aug 24, 2011 at 23:56
  • 1
    $\begingroup$ Thanks, @whuber, this is the answer to the question I'd been trying to ask :). $\endgroup$
    – Andrew Aylett
    Aug 25, 2011 at 16:23
6
$\begingroup$

My instinctive approach may not be the most statistically sophisticated, but you may find it to be more fun :)

I would get a decent-sized sheet of graph paper, and divide up the columns into time blocks. Depending on how long your rides are - are we talking about a mean time of 5 minutes or an hour - you might use different sized blocks. Let's say each column is a block of two minutes. Pick a color for route A and a different color for route B, and after each ride, make a dot in the appropriate column. If there's already a dot of that color, move up one row. In other words, this would be a histogram in absolute numbers.

Then, you would be building a fun histogram with each ride you take, and can visually see the difference between the two routes.

My sense based on my own experience as a bike commuter (not verified through quantification) is that the times will not be normally distributed - they would have a positive skew, or in other words a long tail of upper-end times. My typical time is not that much longer than my shortest possible time, but every now and then I seem to hit all the red lights, and there's a much higher upper-end. Your experience may be different. That's why I think the histogram approach might be better, so you can observe the shape of the distribution yourself.

PS: I don't have enough rep to comment in this forum, but I love whuber's answer! He addresses my concern about skewness pretty effectively with a sample analysis. And I like the idea of calculating in your head to keep your mind off the next hill :)

Improve this answer
$\endgroup$
7
  • 1
    $\begingroup$ +1 For creativity. Actually, your idea is on the path toward practical utility. It would be quite a bit more interesting to use one of the biking tracking sites (I forget which one now, but do add, if you know) to track segment times. If the OP were to come back to CV or StackOverflow with a question about plotting segment time and get a density associated with it, it would be a fabulous statistical exercise - GIS, statistical visualization, and density functions, oh my! :) $\endgroup$
    – Iterator
    Aug 24, 2011 at 23:59
  • 1
    $\begingroup$ I have used Google MyTracks on my phone to track biking segments. I find that the phone is not great at it as it tends to be a power-suck on a device not optimized for it. Garmin (and others) make GPS devices specifically targeted at runners and bikers to track time spent on routes and provide neat charts in an online interface. I don't use a dedicated GPS device myself, but some of my friends use them to share routes on facebook. $\endgroup$
    – Jonathan
    Aug 25, 2011 at 1:23
  • 1
    $\begingroup$ Here is an example of what Garmin device produces. The problem with the charts is that they are already heavily pre-processed, smoothing, etc. Also there is no convenient way to import the data to R for example. But as dedicated device it does its job splendidly, I cannot imagine running or biking without it. $\endgroup$
    – mpiktas
    Aug 25, 2011 at 12:10
  • $\begingroup$ +1 Note that not much skew comes from hitting the red lights (unless they are timed): collectively, they usually only add some Gaussian noise to the time distribution. (Computing its variance is another mental exercise you can do on the next hill.) In practice the skew comes from non-Gaussian variation in the few important factors that control the entire ride: weather, how you're feeling, with whom you're riding, and the occasional accident/detour/traffic jam etc. $\endgroup$
    – whuber ♦
    Aug 25, 2011 at 19:22
  • $\begingroup$ Now that I think about it some more, another very important factor is the time of day. The traffic lights act very differently at peak traffic times - much longer greens for the higher-traffic road. In off-peak times, the lights tend to cycle quickly, defaulting to green for the high-traffic road, but quickly changing when I press the crossing button or a car activates the sensor. $\endgroup$
    – Jonathan
    Aug 25, 2011 at 19:34
5
$\begingroup$

Suppose the two data sets are $X$ and $Y$. Randomly sample one person from each population, giving you $x,y$. Record a '1' if $x > y$ and 0 otherwise. Repeat this many times (say, 10000) and the mean of these indicators will give you an estimate of $P(X_{i} > Y_{j})$ where $i,j$ are randomly selected subjects from the two populations, respectively. In R, the code would go something like: 

#X, Y are the two data sets
ii = rep(0,10000)
for(k in 1:10000)
{
   x1 = sample(X,1)
   y1 = sample(Y,1)
   ii[k] = (x1>y1) 
}

# this is an estimate of P(X>Y)
mean(ii)
Improve this answer
$\endgroup$
2
  • $\begingroup$ This is a good answer, but you could simplify it by removing the for loop: let x1 = sample(X, 10000, replace = TRUE) and y1 = sample(Y, 10000, replace = TRUE) and then calculate mean(x1 > y1) along with mean(x1 == y1) - to get a sense of the # of times the values are equal. $\endgroup$
    – Iterator
    Aug 24, 2011 at 20:55
  • $\begingroup$ Thanks. I knew the loop was unnecessary but I wanted the logic underlying the approach to be abundantly clear. Your code would certainly produce the same results. $\endgroup$
    – Macro
    Aug 24, 2011 at 23:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.