# Bayesian inference - Calculating the prior distribution of the parameter in the Bernouli distribution from a series of bernouli proccesses

What I have are n different time series of bernouli processes of varying lengths, taking the values of 0 or 1. What I would like to do is to use Bayesian inference to calculate, for one of these series, the posterior distribution of the parameter p that drives the proccess. To do this I first need the prior distribution of the Bernouli paramter p which I gather should be available to me through all the n time series that I have. But how do I do this in practice?

To find the mean of the prior distribution of p seems straight forward enough, just count the total number of occurances in all times series and divide by the total number of trials. The variance on the other hand seems a bit less clear to me. I imagine I could take sample means (i.e. frequentist maximum likelihood estimator of p) from each of the n processes and then compute a sampl variance from all of these, is that appropriate? Intuitively it feels not quite right to me since these estimators will themselves not be very indicative of the true value at all (this is in fact the whole reason I am trying to use Bayesian inference here) since they will likely not the long enough. For example I may have one of the n processes be of length ~50 with the true p being something like 0.2.

Is there any way to adress this issue or is the approach I mentioned the best we can accomplish? Will the prior value of the variance affect the results I get in my Bayesian inference later a lot, or is it not so impactful?

Thank you!

Edit: Adding my comment to Tim's answer here to clarify the goal:

I think I should clarify that I do not postulate that all of these Bernoulli Processes have the same p. In fact, I am assuming that they have different values of p. My goal is to find the distribution of p for one specific process (to make things clear we can say that the data available from this process was not used to find the prior). To this end, I need a prior distribution of p, and I'm thinking this one should be found via the other available processes. Does my question make more sense framed like this?

We could for example say that the possible values of p are 0.3, 0.5, and 0.7 all with the same probabilities. If this were the case then we would use that as the prior and then after some observations become more confident that we are dealing with one of them rather than the others. So it is this (unknown) prior distribution that I need to find

• If you believe that many or all of your currently available data may be relevant for continued experimentation, then you can use whatever parts of them you believe to be relevant and with whatever weights you think are appropriate to arrive at a prior distribution for pending experimentation. You are entitled to choose whatever prior you trust most. Of course, you will get better results in the end if your trust is wisely placed. [If you are truly clueless about the relevance of previous data, then it may be best to use a flat or uninformative prior.] Nov 26, 2021 at 19:41

Prior is something you choose prior to looking at your data. So if you want to use a Bayesian model to find the posterior for $$p$$ given your data, you cannot use the same data to choose prior because you would be using this data twice. First, it wouldn't make much sense. If you would be able to estimate the (prior) distribution for $$p$$ from the data, then your problem would be already solved, since your aim was to calculate the distribution of $$p$$. Second, if you used the data to get the prior and then used it for Bayesian inference, you would end up with incorrect (too narrow) uncertainty estimates.

Consider simple example of beta-binomial model. In $$n$$ trials you observed $$k$$ successes, this corresponds to the maximum likelihhod estimate for the probability of success equal to $$k/n$$. Let's say that you used this data to construct a Beta prior for the probability of success $$p$$.

$$p \sim \mathcal{Beta}(\alpha, \beta)$$

with pseudocounts $$\alpha = k$$ and $$\beta = n - k$$ corresponding to $$E[p] = \alpha / (\alpha + \beta) = k/n$$. Given that beta distribution is a conjugate prior for binomial likelihood, we instantly know that the posterior distribution for $$p$$ is

$$p|k,n \sim \mathcal{Beta}(\alpha + k, \beta + n - k)$$

It is the same distribution as the prior, but counting each datapoint twice. With "two times more data" the posterior would be narrower than the prior, since "more data" would make it more precise. You would be cheating yourself.

Prior should be based on what you know about the parameter before collecting your data. So it can be based on theory, some other experiments described in the literature, your wild guess, etc, but not the data.

• Hi, thank you for the answer. I think I should clarify that I do not postulate that all of these Bernoulli Processes have the same p. In fact, I am assuming that they have different values of p. My goal is to find the distribution of p for one specific process (to make things clear we can say that the data available from this process was not used to find the prior). To this end, I need a prior distribution of p, and I'm thinking this one should be found via the other available processes. Does my question make more sense framed like this? I am Nov 26, 2021 at 19:27
• We could for example say that the possible values of p are 0.3, 0.5, and 0.7 all with the same probabilities. If this were the case then we would use this as the prior and then after some observations become more confident that we are dealing with one of them rather than the others. So it is this (unknown) propr distribution that I need to find Nov 26, 2021 at 19:28
• Difficult to see how (discrete) dist'n with .3, .5, .7 equally likely leads to useful prior. Do you really mean 0.301, 0.498, and 0.705 are impossible values of $p?$ // Maybe you're thinking of something like $\mathsf{Beta}(4,4),$ which puts probability mostly in $(,2, .8).$ // One reason for using (continuous) beta priors with binomial likelihoods is that computation of posterior dist'n becomes trivial. [Mybe google 'conjugate prior'.] Nov 26, 2021 at 21:43
• I believe the distribution is continuous, this was simply an example to demonstrate. I want to find a proper continuous prior for this situation. My goal would be to estimate the prior mean and variance from my available data, and then find the corresponding beta parameters from that. Nov 27, 2021 at 6:45

Following on from @Tim's (+1) Answer, it is OK to use prior opinion or data to arrive at what you believe is a reasonable prior distribution for a proposed new experiment.

Beta distributions have support $$(0,1),$$ so they are natural prior distributions for Bernoulli success probability $$p.$$

Suppose you begin with no idea at all about the success probability. Then you might use a flat or non-informative prior distribution such as $$\mathsf{Beta}(1,1)\equiv\mathsf{Unif}(0,1)$$ or the Jeffreys prior $$\mathsf{Beta}(.5, .5).$$

Suppose a preliminary experiment showed $$7$$ successes in $$20$$ trials. Then by Bayes' Theorem the posterior distribution from the uniform prior and that small experiment would be $$\mathsf{Beta}(1+7,\, 1+13)\equiv\mathsf{Beta}(8,14).$$ This (preliminary) posterior has mean $$\mu =$$ $$8/(8+14) =$$ $$8/22 = 0.3636$$ and it puts 95% probability in the (Bayesian credible) interval $$(0.181, 0.570).$$ as computed in R below.

qbeta(c(.025,.975), 8, 14)
 0.1810716 0.5696755


Subsequently, if you decide to do more extensive experimentation to obtain results from $$500$$ additional trials, then you can use the posterior distribution from the preliminary experiment as the prior distribution for the more extensive one (provided, of course, you are still using the same--or very similar--coin or process).

If you get $$x$$ successes in these $$n = 500$$ additional trials, then your updated posterior distribution is $$\mathsf{Beta}(8+x, 14 + n - x).$$ and you can use R as above to get a 95% Bayesian credible interval based on all the information you have.