# How can I estimate the probability of a random member from one population being “better” than a random member from a different population?

Suppose I have samplings from two distinct populations. If I measure how long it takes each member to do a task, I can easily estimate the mean and variance of each population.

If I now hypothesise a random pairing with one individual from each population, can I estimate the probability that the first is faster than the second?

I do have a concrete example in mind: the measurements are timings for me cycling from A to B and the populations represent different routes I could take; I'm trying to work out what the probability is that picking route A for my next cycle will be faster than picking route B. When I actually do the cycle, I've got another data point for my sample set :).

I'm aware that this is a horribly simplistic way to try to work this out, not least because on any given day the wind is more likely to affect my time than anything else, so please let me know if you think I'm asking the wrong question...

-
This is can be done via simple binomial testing & @Macro has a good answer. However, one issue is with the samples themselves: is there anything that could affect your decision to take route A or route B? In particular, do you like to take route A when the roads are dry, the wind is at your back, and dinner's waiting? :) Just be careful of anything that could affect outliers in either set or that could bias the samples in some way. For instance, try setting up your sampling plan ahead of time, with consideration for any need to vary (e.g. safety). –  Iterator Aug 24 '11 at 20:57
One other consideration: Suppose that you have two routes with very similar means and neither dominates the other in terms of the probability that it will be faster. E.g. one is always either 10 or 20 minutes, while the other is always precisely 15 minutes. You may find it better to penalize greater uncertainty (e.g. standard deviation), or to favor one that is more likely to take less than some threshold of time. Your question as-is is fine; I am merely suggesting a future refinement. –  Iterator Aug 25 '11 at 4:32
The statistical question is fine, but if you want to work out the probability which route is faster, I should suggest measuring the lengths of the routes. If the terrain is not hilly then the shorter route will always be faster. –  mpiktas Aug 25 '11 at 11:59
If wind is an important factor, and if the wind speeds are related for the two routes, then it would seem that a person would need information about the dependence between A and B to answer the question accurately. You would need bivariate data for that, and it's hard to ride two paths at the same time. You could enlist someone else to help you collect data, but then would need to account for the variability between riders. In the case A and B are independent, the answers below are great. –  G. Jay Kerns Aug 25 '11 at 16:02
Put another way: if I'm trying to decide which path to take, one's through a tunnel, one's through a field, and the wind's blowing like crazy, I may very well choose the field even if it does horribly worse on average. –  G. Jay Kerns Aug 25 '11 at 16:16
show 1 more comment

### Solution

Let the two means be $\mu_x$ and $\mu_y$ and their standard deviations be $\sigma_x$ and $\sigma_y$, respectively. The difference in timings between two rides ($Y-X$) therefore has mean $\mu_y - \mu_x$ and standard deviation $\sqrt{\sigma_x^2 + \sigma_y^2}$. The standardized difference ("z score") is

$$z = \frac{\mu_y - \mu_x}{\sqrt{\sigma_x^2 + \sigma_y^2}}.$$

Unless your ride times have strange distributions, the chance that ride $Y$ takes longer than ride $X$ is approximately the Normal cumulative distribution, $\Phi$, evaluated at $z$.

### Computation

You can work this probability out on one of your rides because you already have estimates of $\mu_x$ etc. :-). For this purpose it's easy to memorize a few key values of $\Phi$: $\Phi(0) = .5 = 1/2$, $\Phi(-1) \approx 0.16 \approx 1/6$, $\Phi(-2) \approx 0.022 \approx 1/40$, and $\Phi(-3) \approx 0.0013 \approx 1/750$. (The approximation may be poor for $|z|$ much larger than $2$, but knowing $\Phi(-3)$ helps with the interpolation.) In conjunction with $\Phi(z) = 1 - \Phi(-z)$ and a bit of interpolation, you can quickly estimate the probability to one significant figure, which is more than precise enough given the nature of the problem and the data.

### Example

Suppose route $X$ takes 30 minutes with a standard deviation of 6 minutes and route $Y$ takes 36 minutes with a standard deviation of 8 minutes. With enough data covering a wide range of conditions, the histograms of your data might eventually approximate these:

(These are probability density functions for Gamma(25, 30/25) and Gamma (20, 36/20) variables. Observe that they are decidedly skewed to the right, as one would expect for ride times.)

Then

$$\mu_x = 30, \quad \mu_y = 36, \quad \sigma_x = 6, \quad \sigma_y = 8.$$

Whence

$$z = \frac{36 - 30}{\sqrt{6^2 + 8^2}} = 0.6.$$

We have

$$\Phi(0) = 0.5; \quad \Phi(1) = 1 - \Phi(-1) \approx 1 - 0.16 = 0.84.$$

We therefore estimate the answer is 0.6 of the way between 0.5 and 0.84: 0.5 + 0.6*(0.84 - 0.5) = approximately 0.70. (The correct but overly precise value for the Normal distribution is 0.73.)

There's about a 70% chance that route $Y$ will take longer than route $X$. Doing this calculation in your head will take your mind off the next hill. :-)

(The correct probability for the histograms shown is 72%, even though neither is Normal: this illustrates the scope and utility of the Normal approximation for the difference in trip times.)

-
if you have iid realizations from each distribution then what is the advantage of using the normal approximation rather than a monte carlo re-sampling approach (my answer) to estimating $P(X>Y)$? –  Macro Aug 24 '11 at 23:18
@Macro: if the data can be reduced to summary statistics for the Q of interest, one can store less data... just a thought. –  Iterator Aug 24 '11 at 23:25
Sorry, my brain was fried by heat & I missed the obvious answer. You are each answering different questions. The bootstrap method you gave estimates $P(X > Y)$, while @whuber is considering the difference in the mean times, which isn't the same. It isn't too hard to construct a case where option $Y$ is shorter than option $X$ 60% of the time, but the mean for $Y$ is greater than the mean for $X$. –  Iterator Aug 24 '11 at 23:53
FWIW: @whuber is describing Student's t-test for the difference in means between two samples with different standard deviations. –  Iterator Aug 24 '11 at 23:56
Thanks, @whuber, this is the answer to the question I'd been trying to ask :). –  Andrew Aylett Aug 25 '11 at 16:23

My instinctive approach may not be the most statistically sophisticated, but you may find it to be more fun :)

I would get a decent-sized sheet of graph paper, and divide up the columns into time blocks. Depending on how long your rides are - are we talking about a mean time of 5 minutes or an hour - you might use different sized blocks. Let's say each column is a block of two minutes. Pick a color for route A and a different color for route B, and after each ride, make a dot in the appropriate column. If there's already a dot of that color, move up one row. In other words, this would be a histogram in absolute numbers.

Then, you would be building a fun histogram with each ride you take, and can visually see the difference between the two routes.

My sense based on my own experience as a bike commuter (not verified through quantification) is that the times will not be normally distributed - they would have a positive skew, or in other words a long tail of upper-end times. My typical time is not that much longer than my shortest possible time, but every now and then I seem to hit all the red lights, and there's a much higher upper-end. Your experience may be different. That's why I think the histogram approach might be better, so you can observe the shape of the distribution yourself.

PS: I don't have enough rep to comment in this forum, but I love whuber's answer! He addresses my concern about skewness pretty effectively with a sample analysis. And I like the idea of calculating in your head to keep your mind off the next hill :)

-
+1 For creativity. Actually, your idea is on the path toward practical utility. It would be quite a bit more interesting to use one of the biking tracking sites (I forget which one now, but do add, if you know) to track segment times. If the OP were to come back to CV or StackOverflow with a question about plotting segment time and get a density associated with it, it would be a fabulous statistical exercise - GIS, statistical visualization, and density functions, oh my! :) –  Iterator Aug 24 '11 at 23:59
I have used Google MyTracks on my phone to track biking segments. I find that the phone is not great at it as it tends to be a power-suck on a device not optimized for it. Garmin (and others) make GPS devices specifically targeted at runners and bikers to track time spent on routes and provide neat charts in an online interface. I don't use a dedicated GPS device myself, but some of my friends use them to share routes on facebook. –  Jonathan Aug 25 '11 at 1:23
Here is an example of what Garmin device produces. The problem with the charts is that they are already heavily pre-processed, smoothing, etc. Also there is no convenient way to import the data to R for example. But as dedicated device it does its job splendidly, I cannot imagine running or biking without it. –  mpiktas Aug 25 '11 at 12:10
+1 Note that not much skew comes from hitting the red lights (unless they are timed): collectively, they usually only add some Gaussian noise to the time distribution. (Computing its variance is another mental exercise you can do on the next hill.) In practice the skew comes from non-Gaussian variation in the few important factors that control the entire ride: weather, how you're feeling, with whom you're riding, and the occasional accident/detour/traffic jam etc. –  whuber Aug 25 '11 at 19:22
Now that I think about it some more, another very important factor is the time of day. The traffic lights act very differently at peak traffic times - much longer greens for the higher-traffic road. In off-peak times, the lights tend to cycle quickly, defaulting to green for the high-traffic road, but quickly changing when I press the crossing button or a car activates the sensor. –  Jonathan Aug 25 '11 at 19:34

Suppose the two data sets are $X$ and $Y$. Randomly sample one person from each population, giving you $x,y$. Record a '1' if $x > y$ and 0 otherwise. Repeat this many times (say, 10000) and the mean of these indicators will give you an estimate of $P(X_{i} > Y_{j})$ where $i,j$ are randomly selected subjects from the two populations, respectively. In R, the code would go something like:

#X, Y are the two data sets
ii = rep(0,10000)
for(k in 1:10000)
{
x1 = sample(X,1)
y1 = sample(Y,1)
ii[k] = (x1>y1)
}

# this is an estimate of P(X>Y)
mean(ii)

-
This is a good answer, but you could simplify it by removing the for loop: let x1 = sample(X, 10000, replace = TRUE) and y1 = sample(Y, 10000, replace = TRUE) and then calculate mean(x1 > y1) along with mean(x1 == y1) - to get a sense of the # of times the values are equal. –  Iterator Aug 24 '11 at 20:55
Thanks. I knew the loop was unnecessary but I wanted the logic underlying the approach to be abundantly clear. Your code would certainly produce the same results. –  Macro Aug 24 '11 at 23:19