Accounting for uncertain information (few observations) in a prior (empirial Bayes)

Question

I did not really know how to choose an adequate title for this question, so please feel free to change it.

I have a weird case wherein frequentist and Bayesian philosophies come together. I am dealing with a data set of observations on persons who were faced with a situation in which they could either perform an action or could decide not to. The data set consists of several persons with a varying number of observations on that person.

In a frequentist approach, I could say that the probability of a certain person performing that action is the number of times the person performed the action given the situation divided by the number of times the situation occurred. So: if the situation in which the person could perform the action occured $23$ times, and this person has performed the action $8$ times, we could say the probability of this person performing this action in that situation is $\frac{8}{23} = .348$.

Now, when I encounter a new person (not in the data set I already have), I want to have a reasonable estimate of the probability of this person performing this action in the said situation, given very few or no observations at all.

What I do (and what works) is:

• I consider my data set of historical observations and I leave out all persons with less than $x$ observations (in my case: 100 observations).
• I fit a mixture of beta distributions to my data set (since there are different clusters of persons in my data set) using EM.
• I use Bayes' theorem (with a conjugate prior, my fitted Beta distribution is the prior) and update using Bayesian inference to get my probability estimate (and corresponding credible interval) for the newly observed person (of this person performing this action in the said situation).

However, I do not like that I am cutting out all persons with a lower than $x$ number of observations in my prior data set. In my particular case, persons with lower amounts of observations tend to belong to a different cluster compared to persons with a very high amount of observations. So: there is a correlation between the amount of observations and the probability I am trying to calculate! I feel like my prior is biased if I cut out some people (even if they have very few observations and are thus very unreliable).

My question: is there any alternative to my method of cutting out persons with observations below a certain number $x$? Can I maybe somehow still take these persons into account and define their (frequentist) probabilities relative to each other? Can I weigh them into the prior Beta distribution using their credible intervals calculated relative to the other players in the data set? Maybe I should use some sort of regression and factor in another variable for the number of observations..? Or is any other method than the method I am using just impossible?

Answer 1

One of the things I deeply appreciate about Bayes is that if you are not confident in your prior, you can put a prior on your prior.

For example, let's take a simple case where you have just two clusters, one with high observations, one with low. Then:

$p(Action|Obs) = 0\\ +p(Action|Prior_{hi})p(Prior_{hi}|Obs=X)p(Obs=X) \\+p(Action|Prior_{lo})p(Prior_{lo}|Obs=X)p(Obs=X)$

(the zero is just for readability)

$p(Action|Prior_x)$ is, as you already defined it, Beta-distributed. $p(Obs=X)$ is most likely Poisson-distributed; you could assume it, estimate it from data, or if you want a fully online model you could stick ANOTHER prior there - a Gamma distribution - and update it based on the counts you see.

If the boundary between the two was nice and smooth, you could model $p(Prior|Obs)$ as another Beta, where $\alpha_{hi} = beta_{lo} = $ number of cases in the respective cluster with that number of observations and the other parameter - those in the others.

For more clusters, you'd have to go with a more general case of a Categorical-Dirichlet, but otherwise the same logic applies - concentration parameters are preexisting cases of clusters with $X$ observations and some Beta $p(Action|Prior)$ to throw in the blender.

Answer 2

I am sure that there is a highly complex method you could use that I am not aware of, but let me give you a more simplistic approach. I'm going to split this into two parts: philosophical and practical.

Philosophical: It seems like you're struggling to define your population of interest. If the population that you want to make predictions about in the model is only those with a large number of opportunities (which you define as greater than 100), then you're fine the way you are. If your population of interest contains ALL individuals, then throwing out data with low sample size is definitely going to bias your results.

Practical: That being said I can think of two ways to adjust for the low samples size, one frequentest, and one Bayesian. First, the Bayesian. It may be appropriate to essential weight your data. I have to admit, I'm confused why you are using a beta distribution as your prior, as your data follows pretty clearly a Bernoulli distribution with n successes out of x trials. Perhaps I am missing something. I might suggest weighting your data against the number of opportunities that an individual had. I'd need a little more info about your model to be mor specific, but it should be a fairly simple process where you simply make $\sum \alpha + \beta \propto n $ consistently for the contributions of all individuals.

That being said, given your concern about different clusteres, I would personally favor a weighted regression in this case. Here, you could include your other variables as controls, and again weight by the number of opportunities an individuals had. Weighted regression is really easy in most programs too. I've included some sample R code below.

dat<-mydata
mod<-lm(proportion~control.variables,data=dat,weight=no.of.opportunities)

I'm sure others will have more complex suggestions, but there are my thoughts.