# Comparing estimate vs actual frequency

I am working my way through Computer Based Horse Racing Handicapping and Wagering systems: A Report by William Benter.

The following table is presented which shows the public estimate vs actual frequency of the winning horses:

range n exp. act. Z.
0.000 - 0.010  1343  0.007  0.007  0.0
0.010 - 0.025  4356  0.017  0.020  1.3
0.025 - 0.050  6193  0.037  0.042  2.1
0.050 - 0.100  8720  0.073  0.069  -1.5
0.100 - 0.150  5395  0.123  0.125  0.6
0.150 - 0.200  3016  0.172  0.173  0.1
0.200 - 0.250  1811  0.222  0.219  -0.3
0.250 - 0.300  1015  0.273  0.253  -1.4
0.300 - 0.400  716  0.339  0.339  0.0
>0.400  312  0.467  0.484  0.6

n-races = 3198.
n-horses = 32877.
range = the range of estimated probabilities.
n = the number of horses falling within a range.
exp. = the mean expected probability.
act. = the actual win frequency observed.
Z = the discrepancy (+ or -) in units of standards errors.

I am having trouble wrapping my head around what the table is showing.
This is my understanding so far.
Benter calculates an implied probability from the odds of the horse during a given race. Then he puts the horses in different bins depending on the implied probability. For example: there are 1343 horses that have an implied probability x: 0.000 < x <= 0.010 So we expect that from the total population N = 32877 there should be 1343 horses that have a 0.007 chance of winning. So the odds market thinks there are 1343 horses that should have a 0.007 chance of winning. Actual win frequency in that range is 0.007 - but 0.007 of what? The total number of horses?

Sorry if my question is a bit vague - I am struggling with putting in to words what I am missing. Any help is greatly appreciated.

I think you've mostly interpreted the table correctly. Horses are binned by their betting odds of winning, and the "exp" column seems to show the mean probability of winning among the horses in each bin (according to the betting odds). So, for horses that are unlikely to win according to the betting odds (0<p<0.01), they have, on average, odds of ~142:1 (probability of winning ~0.007). This is basically identical to the true odds, as we see that horses in this bin do, in fact, win 1 in 143 races (they actually won 0.7% of their races).

Wins are determined per race, so the 0.007 number indicates that a horse in the first bin has a 0.7% chance of winning its race. Of the 1343 entrants that had a very low implied probability of winning, 9 of them won their race. Note that the same horse could appear in the data multiple times with the same or different odds, so the data points are technically race entrants rather than horses (although "horses" is a reasonable shorthand).

A table like this is typically visualized in a calibration plot, also called a reliability diagram. (If you search for this term, you may want to exclude the term "chemistry", because a "calibration curve" is also used in analytic chemistry, and it's a different thing.)

You start out by binning the interval $$[0,1]$$, as in the "range" column. It's more common to use bins of the same size, e.g., $$0.1$$ or $$0.05$$. You then put the predictions into the bins corresponding to the predicted probabilities of belonging to the target class. (This is important!) Now you have binned your instances.

In the lowest bin $$[0.00,0.01]$$, you have all instances whose predicted probability is at most $$0.01$$. We can calculate the average predicted probability of these instances (which of course must be somewhere between $$0$$ and $$0.01$$). This is given in the "exp." column. We can also calculate which proportion of these instances actually is of the target class. This is given in the "act." column, and this is the answer to your "of what?" question.

If your classifier is correctly calibrated, then these two columns should be identical.

• Thank you for pointing me in the direction of the calibration plot! Apr 15, 2021 at 5:45