How to calculate discrete interval coverage?
What I know how to do:
If I had a continuous model, I could define a 95% confidence interval for each of my predicted values, and then see how often the actual values were within the confidence interval. I might find that only 88% of the time did my 95% confidence interval cover the actual values.
What I don't know how to do:
How do I do this for a discrete model, such as poisson or gamma-poisson? What I have for this model is as follows, taking a single observation (out of over 100,000 I plan to generate:)
Observation #: (arbitrary)
Predicted value: 1.5
Predicted probability of 0: .223
Predicted probability of 1: .335
Predicted probability of 2: .251
Predicted probability of 3: .126
Predicted probability of 4: .048
Predicted probability of 5: .014 [and 5 or more is .019]
Predicted probability of 100 (or to some otherwise unrealistic figure): .000
Actual value (an integer such as "4")
Note that while I've given poisson values above, in the actual model a predicted value of 1.5 may have different predicted probabilities of 0,1,...100 across observations.
I'm confused by the discreteness of the values. A "5" is obviously outside the 95% interval, since there's only .019 at 5 and above, which is less than .025. But there will be a lot of 4's -- individually they are within, but how do I jointly evaluate the number of 4's more appropriately?
Why do I care?
The models I'm looking at have been criticized for being accurate at the aggregate level but giving poor individual predictions. I want to see how much worse the poor individual predictions are than the inherently wide confidence intervals predicted by the model. I'm expecting the empirical coverage to be worse (e.g. I might find 88% of the values lie within the 95% confidence interval), but I hope only a bit worse.