# Estimating probability of success, given a reference population

Suppose you have the following situation:

You observed over time 1000 bowling players, who each played a relatively small number of games (say 1 to 20). You noted the strike percentage for each of those players over the number of games each of those players has played.

A new bowling player comes in and plays 10 games and gets 3 strikes.

The distribution for the number of strikes for any player is assumed to be binomial.

I want to estimate the "true" probability of success for that player.

Please take note of the following:

1. This is not a real situation or a school problem, just a self-thought problem.
2. I am a student with little more stats education than a Stats 101 course. I know a little about inference like maximum likelihood estimation... So feel free to tell me areas in stats which I should read about.
3. My problem might lack information, or if it would be beneficial for, say, the distribution of probability of success to be approximately normal, please tell me so.

Thank you very much

• What do you suppose is the relationship between this player's probability and the probability for each of the other 1000 players? In other words, why would we consider the other 1000 at all in estimating this player's prob.? – rolando2 Jan 21 '12 at 13:37
• I'm assuming that the player's true strike percentage is essentially a realization of the same distribution of strike percentages as the 1000 other players. In other words, there's nothing special about that new player, he's just another random player. Hopefully that makes sense. – Uwat Jan 21 '12 at 18:52

This is a great example to illustrate the difference between frequentist and Bayesian approaches to inference.

My first, simplistic frequentist response: If you've already assumed the distribution of strikes is binomial you don't need to know anything about the other 1000 players (other than perhaps you could use them to check your binomial assumption).

Once you have the binomial assumption clear, your estimate is very straightforward: 3/10. The variance of this estimate is the usual p(1-p)/n = 0.021.

Basically, the 1000 other players are irrelevant unless you think there is something interesting and non binomial about the strike distribution (eg people get better as they play more games).

A more considered Bayesian way of looking at it: Alternatively, if you are interested in applying the prior knowledge you have from other players and you think that the new player is basically a new sample from that same population, you should think of it in Bayesian terms.

Estimate a prior distribution of players. To do this, you need to look at your 1000 data points - the 1000 players who have already been observed, for each of whom you have an estimate of their probability of a strike. Each of these 1000 points can take only one of 21 values (from zero to twenty strikes out of twenty) and you will see a distribution over the whole field. If you convert these scores to proportions (ie between zero and one) this distribution can probably be approximated reasonably well by a probability distribution of a random variable with a Beta distribution. A beta distribution is fully characterised by just two parameters - lets say a and b - but because these parameters are not really to do with the distribution you have asked us about (the particular player's own probability of a strike) but a higher level distribution we call them hyperparameters. You can develop estimates of these hyperparameters from your 1000 data points in one of a number of ways which aren't really relevant to the main point of your question.

Before you have any information on your player at all, your best guess as to his/her proportion of scoring a strike (lets call it p) would just be the most likely value of p from that Beta distribution we just fitted.

However, we have data on our own player, not just the general population! In God we trust, all others must bring data (I'd attribute this quote if I could remember where I found it, sorry). Each time we observe our player play a game and get a strike or not, we have a new piece of information to precisify our estimate of his proportion.

One of the neat things about the beta distribution as a probability distribution for a proportion is that as we gather new information from data and create a new, improved estimate of the proportion, probability theory can show that the new, improved estimate is also a beta distribution - just a more concentrated version. This is because the beta distribution is what is referred to as a conjugate prior when trying to make estimates about a binomial model.

That is, if we observe z out of n successful events (games with strikes in this case); and the prior distribution was beta(a,b); the posterior distribution (are estimate of the probability distribution of p given both the original 1000 data points and are new observation of ten games) is beta(a+z, b+n-z) or (in our case) beta(a+3, b+7). As you can see, the more data you get the less important a and b are. The mathematics of this is reasonably straightforward and in many texts but not that interesting (to me, anyway).

If you have R you can see an example by running the code below (and if you don't have R you should get it - it's free and it's awesome for helping think through this sort of problem). This assumes the prior distribution of players can be modelled by beta(2,5) - this was just made up by me. In reality, there are ways you can estimate figures for a and b better than just making up 2 and 5 because I think the curve looks ok.

As you will see if you run this stylized example, the point estimate of the player's probability of scoring a strike, given a prior distribution of beta(2,5), is 0.29 rather than 0.30. Also, we can create a credibility interval, which is frankly more intuitive and easier to explain than a confidence interval (see many questions and discussions on the internet of the difference between the two, including on CrossValidated).

plot(0:100/100,dbeta(0:100/100,2,5), type="l", ylim=c(0,4), bty="l")
lines(0:100/100,dbeta(0:100/100,2+3,5+7), type="l", lty=2)
legend(0.6,3.5,c("Posterior distribution", "Prior distribution"),
lty=2:1, bty="n")
qbeta(c(0.025, 0.975), 2, 5) # credibility interval prior to any new data
qbeta(c(0.025, 0.975), 2+3, 5+7) # credibility interval posterior to data
qbeta(0.5, 2+3, 5+7) # point estimate of p, posterior to data


Then observe your new player; and calculate a new posterior distribution for the new player. Effectively this says "given what we just observed, where in the distribution of players do we think this person is most likely to be?"

• I don't think this is correct. Suppose that the vast majority (99%) of people among the 1000 people have strike percentage between 5% and 15% and a handful have strike percentages higher than 25%. Then I would argue that it is more likely that the new player we observed has a true strike percentage lower than 30%, but simply "got lucky". – Uwat Jan 21 '12 at 20:16
• ok, good point - I've added an edit to take this situation into account. Basically you have a good statement of a Bayesian inference problem. – Peter Ellis Jan 21 '12 at 21:11
• @Peter - all nicely argued. – rolando2 Jan 22 '12 at 0:18
• Thanks for your answer. However, I did not quite understand what you meant by: "you need an actual distribution of individuals' strike rates which will probably be Beta of some kind" Can you please clarify a little? Thanks – Uwat Jan 24 '12 at 3:45
• Thanks, really good question, I've greatly expanded my answer in response. – Peter Ellis Jan 24 '12 at 9:53