# Mapping categories of probability estimation onto a continuous distribution

I have a model for turning bookmaker odds into probabilities by removing the bookmaker's over-round. This generally works pretty well but I've run into an issue with certain kinds of markets, particularly in a political context. Often the markets will not contain odds of every possible outcome, but will break the possible outcomes down into categories. For example, my model says in relation to an upcoming election that Political Party A has

0.3920% chance of winning 60 seats or fewer
0.5985% chance of winning 61-65 seats
5.0601% chance of winning 66-70 seats
11.6313% chance of winning 71-75 seats
33.2789% chance of winning 76-80 seats
26.7357% chance of winning 81-85 seats
12.9123% chance of winning 86-90 seats
8.2339% chance of winning 91-100 seats
1.1573% chance of winning more than 100 seats

The maximum number of seats that can be won is 151.

I'm interested to map these categories onto a continuous distribution so that I can say stuff like:

"My model's best estimate is that Party A will win XX.XX seats."
"My model says there's a 95% chance that Party A will win between XX.XX-XX.XX seats."
"My model says there's a XX.XX% chance that Party A will win between 72 and 82 seats."

Can anyone suggest how I might go about doing this?

The distribution is discrete rather than continuous, since the number of seats won is an integer in the range $$[0, 151]$$. The problem is that the information you have doesn't fully specify the distribution. Rather, it can be viewed as a set of constraints the distribution must satisfy (more on this below). So, how can we choose a distribution in this underspecified setting?

### Principle of maximum entropy

A reasonable choice follows from the principle of maximum entropy. Loosely, it states that we should choose the distribution that's consistent with existing knowledge, and that that otherwise maximizes our uncertainty. Any other choice would either violate known facts or assume things we don't actually know. More formally, we consider the set of all distributions that satisfy a set of constraints representing our knowledge. From among this set, we choose the distribution with the greatest entropy, which measures uncertainty about the quantity the distribution is defined over. The result is called the MaxEnt distribution for short.

### Defining the distribution

To formalize things, let $$X$$ be a random variable representing the number of seats won. We can represent the distribution of $$X$$ using a vector $$p = [p_1, \dots, p_{152}]^T$$, where $$p_i$$ represents the probability that $$X=i-1$$. So, the first element contains the probability of winning 0 seats, and the last element contains the probability of winning all 151 seats.

### Defining constraints

You have information like "11.6313% chance of winning 71-75 seats". This can be translated to a constraint on $$p$$ as follows. The probability of winning 71-75 seats is given by the sum of elements 72-76 of $$p$$. This sum must equal .11631.

All constraints can be expressed simultaneously by the equation $$C p = v$$, where $$C$$ is a matrix with size $$9 \times 152$$ and $$v$$ is a vector with length $$9$$. Each constraint (of which you have 9) is represented by a row of $$C$$ and the corresponding element of $$v$$. $$C$$ contains a pattern of zeros and ones that specify how to take sums over $$p$$. $$C_{ij} = 1$$ if element $$j$$ of $$p$$ should be included in the sum for the $$i$$th constraint. $$v_i$$ contains the value that this sum must take. For example, the first row of $$C$$ should contain ones in columns 1-61, and the first element of $$v$$ should contain .00392. This represents the first constraint "0.3920% chance of winning 60 seats or fewer".

I'll assume these constraints represent the total available knowledge about the distribution.

### Finding the MaxEnt distribution

The MaxEnt distribution is obtained by solving the following constrained optimization problem:

$$\max_p \ -\sum_i p_i \log(p_i) \quad \text{subject to:}$$

$$C p = v \quad \quad p_i \ge 0 \ \forall i \quad \quad \sum_i p_i = 1$$

The quantity being maximized is the entropy (where we define $$0 \log(0) = 0$$ to avoid undefined values). $$C p = v$$ represents the constraints described in the previous section. The final two constraints simply force the elements of $$p$$ to be nonnegative and sum to one, so that $$p$$ represents a valid probability distribution.

### Result

It might be possible to solve the above optimization problem analytically. But, here's the result of solving it numerically, using a standard optimization algorithm:

As you can see, the MaxEnt distribution is uniform on each of the intervals specified in the constraints. For example, the constraint "33.2789% chance of winning 76-80 seats" translates to 76,77,78,79, and 80 seats each having probability around .066 (given by dividing the total probability assigned to this range by the number of seats it contains).

This is exactly what one would expect, because no information has been given to distinguish the seats in each specified range. And, being maximally uncertain, we should treat them equally. This is a somewhat hand-wavy argument, but I think it should be possible to prove this formally, using the well-known fact that the discrete uniform distribution is the MaxEnt distribution on any finite set of values (in the absence of any further constraints).

### Notes

The result is equivalent to what @PeterFlom suggested (+1). MaxEnt provides a justification for this choice: it's the distribution that's exactly consistent with the information provided, and is otherwise maximally uncertain (i.e. no further assumptions are made). It also gives a principled framework for solving this type of problem in general. If available, additional information and assumptions could be encoded as constraints in the MaxEnt problem, as shown below.

### Imposing further assumptions

Here's an example showing how some further assumptions could be incorporated into the problem. These are not specified in the original question, but might be considered reasonable things to expect.

1. The distribution is unimodal. That is, probability assigned to each number of seats decreases monotonically as we move away from the peak. This is encoded as another set of constraints: $$B p \le \vec{0}$$ (where $$\vec{0}$$ is a vector of zeros). $$B$$ is formatted similarly to $$C$$ above, but contains a pattern of zeros, ones, and negative ones that specify how to take differences. It's constructed such that we have the following system of inequalities. Left of the peak: $$p_1-p_2 \le 0, \ p_2-p_3 \le 0, \dots$$. And right of the peak: $$p_{152}-p_{151} \le 0, \ p_{151}-p_{150} \le 0, \dots$$

2. Probability decreases to zero at 0 or 151 seats. This is simply a pair of constraints: $$p_1 = 0$$ and $$p_{152} = 0$$, which are incorporated into the system $$C p = v$$.

3. Probabilities vary slowly as a function of the number of seats, i.e. the distribution is 'smooth' (speaking loosely, since it's discrete). This assumption is imposed by constraining the squared second differences between the probabilities assigned to adjacent numbers of seats. This constraint is represented by the nonlinear inequality $$g(p) \le \alpha$$, where:

$$g(p) = \sum_{i=2}^{151} \big[ (p_{i+1}-p_i) - (p_i-p_{i-1}) \big]^2$$

$$\alpha$$ is a hyperparameter that controls smoothness; smaller values give smoother solutions. I set it to $$2 \times 10^{-4}$$. If set too low, it can contradict the other constraints, making a solution impossible.

Here's the result of solving the same MaxEnt problem as above (now with the additional constraints):

Clearly, the distribution is smoother. And, although it's hard to see in the plot, the tails taper monotonically to zero rather than remaining flat. I want to emphasize again that this is a consequence of particular assumptions rather than information specified in the problem itself. It's only reasonable to believe the result if the assumptions accurately embody prior knowledge learned from other problems.

• Your answer (+1) is exceptionally clear and I enjoy how it puts @peter-flom’s answer in a new light. I still worry about @whuber’s comment “the uniform approximation could be particularly poor in the tails relative to the information potentially available.” For example, observers would probably think it’s 91 seats is more likely than 100 seats. But I don’t have a clear idea of how much more likely it should be. Is there easy way to encode additional constraints in the MaxEnt problem such that this is taken into account, in some reasonable principled (if approximate) way? – user1205901 - Reinstate Monica Oct 13 '19 at 4:08
• @user1205901 Yes, that's what I had in mind when writing the last section about adding further constraints. It's certainly possible. But, what exactly is this "information potentially available"? If you want tail behavior to be different, you'd have to provide more information about the tails. 91 seats having higher probability than 100 wouldn't be strong enough to affect anything (because this might only mean negligibly higher). – user20160 Oct 13 '19 at 14:49
• @whuber would be best placed to comment on what sort of information he had in mind, though I welcome comments from any user who can think of information that might be relevant. The only extra information I can think of is that which comes from observers (of which, I guess, I am one) who have domain-specific knowledge in this area of politics. They would doubtless think 91 seats is much more likely than 100, but any quantification of how much more likely would necessarily be a guess of some kind. – user1205901 - Reinstate Monica Oct 14 '19 at 0:24
• @user1205901 I edited the answer to include an example showing how to incorporate some additional assumptions. The tails behave in a way you might like, based on your comments (but see notes at the end). Whether or not the particular result is suitable, it can at least be considered a demonstration of the general method. – user20160 Oct 14 '19 at 5:13
• @user1205901 Glad to help. I'm not familiar with the MaxEnt toolbox. The documentation seems to say it's designed for distributions over binary vectors, which wouldn't be appropriate for this problem. I'll try to put some code up in a bit. If you want to try things in the meantime, I just used matlab's fmincon() to solve the optimization problems described in the post. – user20160 Oct 15 '19 at 5:28

You will have to make some assumptions about the distribution of the probabilities within each category. The simplest is to use a uniform distribution, so, e.g. turn "12.91 of 86-90" into:

86 - $$12.91/5 = 2.58$$

87 - $$12.91/5 = 2.58$$

88 - $$12.91/5 = 2.58$$

89 - $$12.91/5 = 2.58$$

90 - $$12.91/5 = 2.58$$

This isn't exactly realistic, but it probably doesn't make too much difference. A more realistic (but much more complex) method would use the overall distribution to estimate the probabilities within each category. This could get complicated if the distribution is odd.

Just eyeballing what you presented, a Normal distribution seems reasonable. So, you can find the mean and sd of the data you've got and then use a normal distribution with those parameters. There are lots of online calculators to get mean and sd of grouped data (these may rely on some assumptions of their own).

• Thanks! Sorry to ask a couple of follow-up questions, but 1. Would you be able to clarify a bit further about how the overall distribution could be used to estimate the within-category probabilities? 2. Regarding the use of a normal distribution, I found a calculator at ncalculators.com/statistics/… and generated the following output: i.imgur.com/STBW8Ob.png, which suggests a mean of 81.25 and an SD of 8.91. But I wondering whether it's correct for me to use the probabilities as frequencies in this context? – user1205901 - Reinstate Monica Sep 26 '19 at 22:55
• 1. You would modify what I did for the uniform distribution to match the overall distribution within the categories. 2. I don't understand what you mean by "probabilites as frequencies". – Peter Flom Sep 27 '19 at 13:53
• With data this extensive I would be concerned that the uniform approximation could be particularly poor in the tails relative to the information potentially available. This distribution obviously is positively skewed, precluding the use of a Normal fit (unless the data are suitably transformed first). Note, too, that the distribution is discrete and this could be an important factor. For instance, suppose party $A$ holds a majority once they attain 96 or more seats: what are the chances? The uniform fit could easily be off by a factor of two or more with such questions. – whuber Sep 27 '19 at 14:43
• @PeterFlom By "probabilities as frequencies" I meant that the calculator I linked (and other similar online calculators for getting the mean and sd of grouped data) requests a column of frequencies, e.g. the frequency between 66-70. I don't have any frequencies and so I just used the percentage probability instead (i.e. 5.0601). – user1205901 - Reinstate Monica Sep 27 '19 at 22:04
• @whuber Under the circumstances, what do you think the best way forward would be? You are right that the probability Party $A$ will hold a majority is a question of interest. Just in case it matters I'll mention that for a majority they need 76 or more seats, rather than 96. – user1205901 - Reinstate Monica Sep 27 '19 at 22:08