14
$\begingroup$

I was studying some statistics in DataCamp and they assigned me this exercise that I can't solve. I tried speaking with people that know more statistics than me and we can't seem to agree in an answer. The exercise is as follows:

Since 1926, the Belmont Stakes is a 1.5 mile-long race of 3-year old thoroughbred horses. Secretariat ran the fastest Belmont Stakes in history in 1973. While that was the fastest year, 1970 was the slowest because of unusually wet and sloppy conditions. With these two outliers removed from the data set, compute the mean and standard deviation of the Belmont winners' times. Sample out of a Normal distribution with this mean and standard deviation using the np.random.normal() function and plot a CDF. Overlay the ECDF from the winning Belmont times. Are these close to Normally distributed?

The answer to this is yes, they are. I used the dataset they provided and calculated both the theoretical CDF and the ECDF. They match.

Unfortunately, Justin was not alive when Secretariat ran the Belmont in 1973. Do you think he will get to see a performance like that? To answer this, you are interested in how many years you would expect to wait until you see another performance like Secretariat's. How is the waiting time until the next performance as good or better than Secretariat's distributed? Choose the best answer.

The thing is, they say the distribution is exponencial, and I understand it. We can model the event by a Bernoulli trial, where we toss a coin that says if we see a fast horse or not in a given year. Knowing that the probability of seing such horse is small, this turns into a Poisson process and the time between poisson events is exponentially modelled.

My problem with this answer is that we only have a race per year and the time should be continuous to use the exponential distribution. Also, even though the probability of seing a fast horse is small, the number of trials is not high, so I don't even know why we can modell it as a Poisson process to begin with.

I appreciate your help!

$\endgroup$
7
  • 1
    $\begingroup$ "Normally" does not mean "exponential"! With a Normal distribution of winning times--and, pace @Glen_b, that's not an unreasonable model provided the standard deviation is small relative to the mean time and the fit to the data is good--the process is not Poisson and the waiting times are not exponential. $\endgroup$
    – whuber
    Aug 30, 2022 at 10:13
  • 2
    $\begingroup$ BTW, the data at belmontstakes.com/history/past-winners indicate a downward trend from 1926 to c. 1970 - 1990. The residuals relative to a robust smooth do indeed look Normal, apart from the two outliers mentioned. Returning to your question, where in the original exercise is it suggested that you should model this time series as a Poisson process? $\endgroup$
    – whuber
    Aug 30, 2022 at 10:32
  • 1
    $\begingroup$ It is always possible that the length of the Belmont Stakes may change during Justin's lifetime and so a shorter time might be observed. Indeed precisely this did happen in 2020, when Tiz the Law won with a time 1:46.53 over 1.125 miles (the same distance as 1893 and 1894) compared to Secretariat's 2:24 over the standard 1.5 miles. $\endgroup$
    – Henry
    Aug 30, 2022 at 15:04
  • $\begingroup$ @Glen_b to be fair to DataCamp, the did not say that the data is normally distributed, they ask me to verify if it is and I did this by code. I just thought that it would not be relevant to mention this here. $\endgroup$ Aug 30, 2022 at 22:31
  • 1
    $\begingroup$ Ok, I think I know that the problem is: I did not know that I "quoted" them. For me, this was just a way to highlight text, not to quote. I'm sorry and I eddited the question. $\endgroup$ Aug 30, 2022 at 23:12

1 Answer 1

23
$\begingroup$

Let's begin by answering the question as it stands. Then we can respond to some of the points raised in comments.

The question wants you to make the following assumptions:

  1. The future will behave like the past, but "outliers" like Secretariat will not occur.

  2. The past results are characterized as a sequence of realizations of random variables that are

    • independent
    • identically distributed (thus, exhibit no trends over time)
    • normal
  3. Your estimate of that common normal distributions is perfect -- there is no error in it.

With these assumptions, let $\mu$ be the common Normal mean, $\sigma$ the common Normal standard deviation, and $x_{min}$ Secretariat's time (144 seconds). The chance of equaling or beating that time on each future run of the Belmont stakes therefore is the chance that a standard Normal variate $Z$ will be less than or equal to $p = (x_{min}- \mu)/\sigma,$ given by the standard Normal distribution function $\Phi.$

This describes a sequence of independent identically distributed Bernoulli$(p)$ variables. The chance that the waiting time $T$ exceeds $N$ runs of the stakes (its survival function) is the chance the next $N$ values equal $0.$ That is the product of those chances (by the independence part of assumption $(2)$), given by

$$\Pr(T \gt N) = (1-p)^{N}.$$

This is a geometric distribution. However, $p$ is tiny. (It will be somewhere around $10^{-3},$ $10^{-4},$ or even less, depending on how you estimate $\mu$ and $\sigma.$) Thus, to an excellent approximation,

$$\Pr(T \gt N) = (1-p)^{N} = ((1-p)^{1/p})^{pN} \approx \exp(-pN).$$

That is manifestly an exponential waiting time, consistent with your expectation that Poisson processes might play a role. The expected waiting time is

$$E[T] = \sum_{N=0}^\infty \Pr(T \ge N) = \frac{1}{p} = \frac{1}{\Phi\left((x_{min}-\mu)/\sigma\right)}.$$

That will be hundreds to tens of thousands of runs (and therefore at least that many years). Bear in mind that the assumptions $(1)$ - $(3)$ must continue to hold for at least Justin's lifetime for this to be a useful calculation.


Let's do a reality check. The data at https://www.belmontstakes.com/history/past-winners/ give 94 winning times from 1926 through the present. Suppose Justin expects to live another $N$ years. Assume only the first two parts of $(2)$ -- namely, winning times are independent and identically distributed. In that case, the location of the best time during the entire time series of $94 + N$ results is randomly and uniformly distributed: it could occur at any time. Consequently, Justin would compute a probability of $N/(94 + N)$ of observing the best time during the next $N$ years. With this argument, they would have a greater than 50% chance of observing the best time provided $N$ exceeds $94.$ This is a couple of orders of magnitude shorter than the previous result.

Which should we believe?

We might appeal to the data.

enter image description here

This plot immediately shows most of the assumptions $(1) - (3)$ are implausible or must be modified. There have been trends, leveling off around 1970 - 1990. Consequently, "$94$" in the preceding calculations ought to be replaced by some value between c. $32$ and $52.$ If Justin is young, they should expect to see a new record time during their lifetime.

Relative to these trends, the times do display remarkably consistent randomness, as suggested by this time series plot of the residuals (differences between the winning times and their smoothed values):

enter image description here

Moreover, there is no significant evidence of lack of independence: the autocorrelations are negligible.

enter image description here

This justifies examining the univariate distribution of the residuals. Indeed, it is well described as a Normal distribution (the red curve), provided we ignore two of the $94$ data points.

enter image description here

These residuals (after removing the two extreme values) have a mean of $-0.015$ and a standard deviation of $1.268.$ The residual for the best time is $-4.732$ (almost five seconds better than expected that year based on the Loess fit). The intended answer to the question, then, is tantamount to a mean waiting time of

$$E[T] = \frac{1}{\Phi\left((4.732-(-0.015))/1.268\right)} \approx 10^4.$$

The Belmont Stakes will never run that many times.

The post-Secretariat residuals have a bit more scatter than the older residuals, with a standard deviation of $1.5.$ An upper $95\%$ confidence limit for this estimate is $2.0.$ Using that instead of $1.268$ in the preceding calculation causes the estimated waiting time to drop two orders of magnitude from $10\,000$ to $100.$ That's still a long time, but it leaves some room for Justin to hope! This shows how sensitive the answer is to what otherwise seems to be a minor technical issue, that of estimating the Normal parameters.


That still leaves several thorny issues:

  • How should we treat the fact that the parameters of this distribution can only be estimated and are not very certain?

    For instance, the solution is exquisitely sensitive to the estimate of $\sigma$ and that could be off by $30\%$ or more. We can do this with the Delta method or bootstrapping. You can read about these elsewhere on CV.

  • What justifies ignoring the two "outliers"?

    The correct answer is, nothing. "Outlying" conditions will recur in the future -- one can almost guarantee that. Since $2$ outliers have appeared in $94$ runs, we can predict (with 95% confidence) that between $0$ and $8$ will appear in the next $94$ runs, suggesting Justin has a chance of observing several unusual winning times (good or bad). (The value $8$ is a nonparametric $95\%$ upper prediction limit.)

  • Why can we assume that trends -- which clearly occurred in the past -- won't recur in the future?

    For instance, it's difficult to believe that gradual improvements halted a third of a century ago in a competitive business like horse racing. An alternative is that some countervailing effects may have slowed progress. Maybe gradual warming of the climate adversely affects winning times? Only a tiny, tiny effect is needed. If the warming accelerates, the future curve might trend upwards. That would make it less likely for any horse ever to beat Secretariat's time.


The moral of this post is that data analysis is not a matter of hiring a roomful of monkeys (pardon me, DataCamp graduates) to plug numbers into Poisson process calculators that spit out predictions and probabilities. Data analysis, when practiced well, is a principled exploration of how data help us reason from assumptions to tentative conclusions. As this case study shows, even slight differences in in the assumptions can lead to profound differences in the conclusions. The value of a data analysis, then, lies not in its predictions but primarily in the care with which it is conducted, how well it exposes and examines the assumptions, and how it reveals their connections to the results.

$\endgroup$
6
  • 4
    $\begingroup$ The last para... The moral. These should be in some sort of oath lines taken by a data scientist. Another amazing post, whuber. +1. $\endgroup$ Aug 30, 2022 at 14:38
  • 1
    $\begingroup$ +1. Very nice analysis and reasoning! How did you calculate the nonparametric prediction interval for the number of outliers? $\endgroup$ Aug 30, 2022 at 14:53
  • 2
    $\begingroup$ @COOL The first $n$ observations out of $n+N$ present and future iid observations is a sample without replacement from all $n+N.$ If you observe $k$ outliers in the first $n$ and there are $X$ more in the remaining $N,$ then the UPL is the smallest integer $t$ for which $\Pr(X \le t) \lt 1 - 0.95.$ This is a hypergeometric (right tail) probability for sampling a population in which $k+X$ objects are "nonconforming." A quick search found $t=8:$ with(list(t = 0:8, n = 94, N = 94, k = 2), phyper(k, m, n, k + t)) The last value is almost exactly 5%. $\endgroup$
    – whuber
    Aug 30, 2022 at 15:09
  • $\begingroup$ @whuber Got it, thanks! I think it's the same approach explained in Meeker & Hahn (section 6.7.1). $\endgroup$ Aug 30, 2022 at 15:22
  • $\begingroup$ @COOL Most likely, because that is one of the places I learned nonparametric stats and is still a key resource when I need to look things up. (The other was the literature in environmental monitoring in the late 80's and early 90's.) $\endgroup$
    – whuber
    Aug 30, 2022 at 15:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.