4
$\begingroup$

In a project I am working on I need to persuade people of the truth of a little story I made up. It would be easier though if I could tell them some key phrase that they could then look up on Wikipedia or some such...

So the story...

Say that you were a golfing coach that was training a collection of young golfers to enter next year's World Hole In One championships. The format of the contest is that each entrant would have 1000 shots and the winner is the one that got the most holes in one. Judging by previous years, the winner would get perhaps two or three.

Your task is to select which one of your young players you should select to send to the contest as only one is allowed. Now you can use one of two methods to choose your entrant.

Method A. Let all of your young golfers play 1000 shots and then select the golfer with the highest number of holes-in-one (probably of the order of 1 or 2).

Method B. Let all your golfers play just 100 shots and select the one with the smallest average distance from the hole.

It is my feeling that the players selected using method B would stand a better chance of winning the world hole in one championships than the golfer selected by method A. This may appear paradoxical to some people because I am valuing a "proxy" for winning over and above the "real" test. Assuming I am correct (and feel free to tell me I'm wrong), is there some name I could give to this effect? Or a chapter in a book that tells an analogous tale.

$\endgroup$
  • $\begingroup$ The chance of hitting a hole in one is related to the final lie in a complex fashion. Consider (hypothetically) what the outcomes would have been had the hole only been painted on the green: those who hit holes in one likely would have had lies away from the hole. Hitting a hole in one means that the path of the rolling ball intersected the hole (at a time when the ball had sufficiently low momentum to drop in). Thus, neither of the methods looks particularly useful for selecting the competitors, because A collects insufficient data and B's data aren't directly relevant. $\endgroup$ – whuber Oct 24 '19 at 19:37
  • 2
    $\begingroup$ I think @whuber has clearly expressed some of what I was trying to say in my answer. I don't completely agree that neither method is "useful" however. As an extreme example: I certainly have a lower hole-in-one probability than Tiger Woods... but given limited balls hit it's not entirely impossible (wishful thinking, perhaps) that I get lucky. On the other hand, I can safely guarantee that Tiger will routinely be "closer to the hole" than me even with a very small number of balls hit. $\endgroup$ – knrumsey Oct 24 '19 at 20:24
  • $\begingroup$ This question desperately needs a more informative title $\endgroup$ – Jake Westfall Oct 26 '19 at 15:33
8
$\begingroup$

This is the problem of rare event estimation. The number of shots made by player $i$ is a Binomial random variable $$X_i \sim Binom(1000, p_i)$$ where $p_i \approx 1/1000$ is very small. In general, $1000$ balls is not nearly enough to accurately estimate $p_i$. The problem is that the likelihood function has too much "overlap" for different values of $x_i$.

enter image description here

If Method A is used, you are "most" likely to select the player with the highest $p_i$, but $1000$ shots is not enough to select this player with high probability.


Here is a simple Monte Carlo simulation (in R) illustrating this point. I am considering $10$ hypothetical players with success probabilities evenly spaced between $1/1000$ and $3/1000$.

num_players <- 10
num_shots <- 1000
probs <- seq(1/1000, 3/1000, length.out=num_players)
M <- 10000 #Number of monte carlo simulations

#Monte Carlo simulation
player_selected <- rep(NA, M)
for(i in 1:M){
  shots_made <- rbinom(num_players, num_shots, probs)
  player_selected[i] <- which.max(shots_made)
}
round(table(player_selected)/sum(table(player_selected))*100,2)

Here is the results, showing the percentage of time each player was selected. The best player was only chosen by Method A $\approx 18\%$ of the time and alarmingly, one of the $5$ worst players was chosen $27\%$ of the time.

1     2     3     4     5     6     7     8     9    10 
2.37  3.25  5.25  7.26  8.98 10.58 12.86 14.97 16.84 17.64 

Method B

You are assuming that "probability of hole-in-one" is related to (correlated with) "distance from hole". This is probably true, but hard to quantify/model.

Formally, you propose estimating a proxy $\theta_i$ which is hopefully correlated to $p_i$ with the hope of getting better "separation" of the likelihood functions. Although this may lead to better results, there is no guarantee that this will work better and depends on your modeling assumptions.

For example (and I am not claiming this is a good modeling choice), assume that the distance from the hole for player $i$ is a half-Normal random variable with scale parameter $$\sigma_i = -\log(c p_i)$$

A similar simulation study can be conducted.


scales <- -log(c*probs)
#Monte Carlo simulation
player_selected <- rep(NA, M)
for(i in 1:M){
  shot_distance <- abs(rnorm(num_players, 0, scales/sqrt(1000)))
  player_selected[i] <- which.min(shot_distance)
}
round(table(player_selected)/sum(table(player_selected))*100,2)

If $c = 500$, then Method B is an improvement over Method A.

1     2     3     4     5     6     7     8     9    10 
2.52  3.07  3.56  4.31  5.10  6.33  8.73 12.40 18.48 35.50 

If $c=50$, then Method B is actually worse than Method A.

1     2     3     4     5     6     7     8     9    10 
7.51  7.95  8.76  9.18  9.56 10.19 11.02 11.47 11.87 12.49 
$\endgroup$
  • $\begingroup$ Thanks for the detailed answer! If I'm understanding you correctly, the best approach would probably entail recording the distance-to-hole for thousands and thousands of shots and trying to learn the underlying distribution. Then you'd know which approach is better. $\endgroup$ – Andrew Brēza Oct 24 '19 at 19:19
  • 2
    $\begingroup$ You are assuming that "probability of hole-in-one" is related to (correlated with) "distance from hole". This is probably true, but hard to quantify/model. $\endgroup$ – knrumsey Oct 24 '19 at 19:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.