Bayesian inference - Calculating the prior distribution of the parameter in the Bernouli distribution from a series of bernouli proccesses

Question

What I have are n different time series of bernouli processes of varying lengths, taking the values of 0 or 1. What I would like to do is to use Bayesian inference to calculate, for one of these series, the posterior distribution of the parameter p that drives the proccess. To do this I first need the prior distribution of the Bernouli paramter p which I gather should be available to me through all the n time series that I have. But how do I do this in practice?

To find the mean of the prior distribution of p seems straight forward enough, just count the total number of occurances in all times series and divide by the total number of trials. The variance on the other hand seems a bit less clear to me. I imagine I could take sample means (i.e. frequentist maximum likelihood estimator of p) from each of the n processes and then compute a sampl variance from all of these, is that appropriate? Intuitively it feels not quite right to me since these estimators will themselves not be very indicative of the true value at all (this is in fact the whole reason I am trying to use Bayesian inference here) since they will likely not the long enough. For example I may have one of the n processes be of length ~50 with the true p being something like 0.2.

Is there any way to adress this issue or is the approach I mentioned the best we can accomplish? Will the prior value of the variance affect the results I get in my Bayesian inference later a lot, or is it not so impactful?

Thank you!

Edit: Adding my comment to Tim's answer here to clarify the goal:

I think I should clarify that I do not postulate that all of these Bernoulli Processes have the same p. In fact, I am assuming that they have different values of p. My goal is to find the distribution of p for one specific process (to make things clear we can say that the data available from this process was not used to find the prior). To this end, I need a prior distribution of p, and I'm thinking this one should be found via the other available processes. Does my question make more sense framed like this?

We could for example say that the possible values of p are 0.3, 0.5, and 0.7 all with the same probabilities. If this were the case then we would use that as the prior and then after some observations become more confident that we are dealing with one of them rather than the others. So it is this (unknown) prior distribution that I need to find

Answer 1

Prior is something you choose prior to looking at your data. So if you want to use a Bayesian model to find the posterior for $p$ given your data, you cannot use the same data to choose prior because you would be using this data twice. First, it wouldn't make much sense. If you would be able to estimate the (prior) distribution for $p$ from the data, then your problem would be already solved, since your aim was to calculate the distribution of $p$. Second, if you used the data to get the prior and then used it for Bayesian inference, you would end up with incorrect (too narrow) uncertainty estimates.

Consider simple example of beta-binomial model. In $n$ trials you observed $k$ successes, this corresponds to the maximum likelihhod estimate for the probability of success equal to $k/n$. Let's say that you used this data to construct a Beta prior for the probability of success $p$.

$$ p \sim \mathcal{Beta}(\alpha, \beta) $$

with pseudocounts $\alpha = k$ and $\beta = n - k$ corresponding to $E[p] = \alpha / (\alpha + \beta) = k/n$. Given that beta distribution is a conjugate prior for binomial likelihood, we instantly know that the posterior distribution for $p$ is

$$ p|k,n \sim \mathcal{Beta}(\alpha + k, \beta + n - k) $$

It is the same distribution as the prior, but counting each datapoint twice. With "two times more data" the posterior would be narrower than the prior, since "more data" would make it more precise. You would be cheating yourself.

Prior should be based on what you know about the parameter before collecting your data. So it can be based on theory, some other experiments described in the literature, your wild guess, etc, but not the data.

Answer 2

Following on from @Tim's (+1) Answer, it is OK to use prior opinion or data to arrive at what you believe is a reasonable prior distribution for a proposed new experiment.

Beta distributions have support $(0,1),$ so they are natural prior distributions for Bernoulli success probability $p.$

Suppose you begin with no idea at all about the success probability. Then you might use a flat or non-informative prior distribution such as $\mathsf{Beta}(1,1)\equiv\mathsf{Unif}(0,1)$ or the Jeffreys prior $\mathsf{Beta}(.5, .5).$

Suppose a preliminary experiment showed $7$ successes in $20$ trials. Then by Bayes' Theorem the posterior distribution from the uniform prior and that small experiment would be $\mathsf{Beta}(1+7,\, 1+13)\equiv\mathsf{Beta}(8,14).$ This (preliminary) posterior has mean $\mu = $ $8/(8+14) = $ $8/22 = 0.3636$ and it puts 95% probability in the (Bayesian credible) interval $(0.181, 0.570).$ as computed in R below.

qbeta(c(.025,.975), 8, 14)
[1] 0.1810716 0.5696755

Subsequently, if you decide to do more extensive experimentation to obtain results from $500$ additional trials, then you can use the posterior distribution from the preliminary experiment as the prior distribution for the more extensive one (provided, of course, you are still using the same--or very similar--coin or process).

If you get $x$ successes in these $n = 500$ additional trials, then your updated posterior distribution is $\mathsf{Beta}(8+x, 14 + n - x).$ and you can use R as above to get a 95% Bayesian credible interval based on all the information you have.