Beta binomial Bayesian updating over many iterations

Question

I'm using a beta binomial updating model for a piece of code that I am writing. The software is real time updating - meaning that data is continually being gathered and after N data points are gathered, the bayesian model is updated using the N data points.

Under this logic, I am using the posterior output as my prior for the next iteration. My problem is that over billions/trillions/maybe more of iterations, the bayesian beta parameters (alpha and beta) will grow very large. I am worried that eventually the parameters will become so large that they will cause an integer overflow in memory.

So my question is twofold -

1. Is it reasonable to be worried about this integer overflow. I understand that $2^{32}$ is an extremely large number, but I'm building this software for an internet service that will be running 24/7, 365 days a year and I don't want it to crash. For example if I was updating it with 1,000,000 data points a day then the model would only last ~4000 days before an integer overflow.

2. Is it possible to transform a Beta(x,y) r.v., where x and y are extremely large, to a Beta(x*,y*) r.v. where x* and y* are relatively smaller? The transformed Beta doesn't have to be exact, just similar.

Answer 1

1) You could scale it down, so $\alpha,\beta\mapsto \alpha/N, \beta/N$. This would indeed allow you to continue. What this would do, however, is to make older data carry less weight (if $N$ is two, it would be carrying half as much weight). This might even be a feature, if you would rather trust newer data.

Compare for example $\alpha=\beta=20$ and $\alpha=\beta=10$ here. What you are doing when dividing by $N$ is multiplying the variance of the distribution by $N$ (almost!) while leaving the expected value unaffected.

2) You could stop right there. With 1 million data points, you distribution will essentially be a point. If you are having troubles with your model, despite 1000000 data points, you don't need more data, you need a better model.

In short, overflow shouldn't be a problem with a binomial-beta setup, because long before you reach overflow, you will have insanely small confidence intervals.

Answer 2

If you continue to update your prior in the manner that you described, aren't you assuming that the process that is generating your data stationary?

If the answer to the question is yes, then all that you should need to do is take a random sample of your data to create a likelihood function and then generate the posterior. In that way you would not have to worry about overflow.

On the otherhand, although I do not know what the process is that you are investigating, it seems almost impossible that a process could remain stationary over any long period of time. In fact, you could check to see if your data generating process is serially changing by monitoring independent estimates of the alpha and beta parameters over time. Minimally, you could make a control chart of the two parameters; or better yet there is probably a simple way to implement a likelihood ratio to check for stationarity.

Answer 3

If alpha and beta are very large, your prior distribution must have converged already to a single point, and you can use the MAP approximation instead of the posterior distribution.

Having said that, scaling alpha and beta down would preserve the mean and keen you away from conversion (if that's what you're looking for).

See python code:

from conjugate_prior import BetaBinomial
heads = 95
tails = 105
prior_model = BetaBinomial() # Uninformative prior
updated_model = prior_model.update(heads, tails)
credible_interval = updated_model.posterior(0.45, 0.55)
print ("There's {p:.2f}% chance that the coin is fair".format(p=credible_interval*100))
predictive = updated_model.predict(50, 50)
print ("The chance of flipping 50 Heads and 50 Tails in 100 trials is {p:.2f}%".format(p=predictive*100))
scaled_down_model = BetaBinomial(BetaBinomial.mean()) # preserve mean, new model

Answer 4

The approach I found useful for this is to divide the a and b parameters by the maximum value of the y axis at each iteration. Thus keeping the scale constant.