6
$\begingroup$

Normally you have a beta distribution with shape parameters $a$ and $b$. The mean of this distribution is $a / (a + b)$ and the sample size, or the confidence (or K) is $a + b$.

Now, if you do some trials, with let's say N positive outcomes and M negative outcomes, you end up with a posterior distribution that is $\text{Beta}(a + N,~b + M)$. So, now your mean is $\frac{a + N}{a + N + b + M}$ and the sample size / confidence / K is $a + N + b + M$.

Now, my question is: what if you want to keep the K / confidence level fixed? So let's say K should always be 10. So $\text{Beta}(5,5)$ is fine, as is $\text{Beta}(9,1)$, or $\text{Beta}(1.23, 8.77)$.

In other words: in this case I would like the mean of the posterior to be able change to reflect the evidence found in the new data, but the confidence level should remain the same (rather than increasing all the time).

Is there a simple update rule for this scenario as well?

$\endgroup$
4
  • 2
    $\begingroup$ You appear to be using the term "confidence" in an unusual sense. Could you provide us a definition, please? $\endgroup$
    – whuber
    Apr 2 '13 at 17:42
  • $\begingroup$ Hi, thanks! What I mean is (a + b). So normally this would be referred to as the sample size, I guess. But I thought a different interpretation was the amount of confidence you have in the distribution at hand. So high K means high confidence (i.e. you need a whole lot of new examples to alter your beliefs as reflected by the distribution). $\endgroup$
    – Tom
    Apr 3 '13 at 8:02
  • 1
    $\begingroup$ And I am referring to it as K because I am reading John Kruschke's (excellent) "Doing Bayesian Data Analysis" in which it has that name... $\endgroup$
    – Tom
    Apr 3 '13 at 8:07
  • 1
    $\begingroup$ So please check my understanding: given your assumed prior is Beta$(N,M)$, the sample size is $a+b$, and $K=a+N+b+M$, you would like to know how to "keep ... $K$ ... fixed." Isn't the answer to set the sample size to $K-(N+M)$? $\endgroup$
    – whuber
    Apr 3 '13 at 14:00
2
$\begingroup$

I've taken a look at the book. It seems to me that the rationale for this "prior sample size" teminology is the following. We have the usual model with $X_1,\dots,X_n$ conditionaly independent and identically distributed, given $\Theta=\theta$, with distribution $X_1\mid\Theta=\theta\sim\mathrm{Ber}(\theta)$. Suppose that a priori $\Theta\sim\mathrm{Beta}(a,b)$. The prior mean is just $\mathbb{E}[\Theta]=a/(a+b)=:\mu$. If $a$ and $b$ are integers bigger than $1$, one way to interpret this prior is to suppose that we started with a $\mathrm{U}[0,1]$ prior and observed $a-1$ successes and $b-1$ failures in a Bernoulli experiment. By Bayes's theorem, the "posterior" of $\Theta$ for this gedanken experiment is exaclty $\mathrm{Beta}(a,b)$. Hence, we may suggestively define the prior sample size $\nu:=a+b-2$. We know from the properties of the beta distribution that a bigger $\nu$ will give us a more concentrated distribution. Now, to answer your question, what you want to do seems impossible: the more data you observe, the smaller will be your posterior uncertainty about $\Theta$. The posterior of $\Theta$ is $\mathrm{Beta}(c,d)$, with $c=a+\sum_{i=1}^n x_i$, and $d=b+n-\sum_{i=1}^n x_i$. Hence, $c+d$ grows linearly with $n$.

$\endgroup$
3
  • $\begingroup$ that is right. Normally, the more data you observe, the more certain you get of your posterior estimations. That is quite exactly what it is all about ;-) However I want to use this in an environment where posterior distribution you are trying to estimate is not stable, but rather changing over time. So let's say you are trying to estimate the overall bias of a factory producing coins (to stick to the familiar coin flipping scenario ;-) ). All you get is coins from the factory produced that day. And the point is that bias of the factory might gradually change over time. $\endgroup$
    – Tom
    Apr 4 '13 at 8:06
  • $\begingroup$ @Tom: The word for that is a "nonstationary process". $\endgroup$
    – Neil G
    Apr 9 '13 at 21:02
  • $\begingroup$ OK, I didn't know there was a term for it. So that helps ;-) Thanks! $\endgroup$
    – Tom
    Apr 10 '13 at 8:20
2
$\begingroup$

I like Zen's answer and want to add to it in order to clarify some misconceptions in the question. "Confidence" or "sample size" cannot be meaningfully applied to a posterior distribution. Only a likelihood can have a notion of "sample size". That's the motivation behind Zen's point that the confidence should be $a + b - 2$.

In general, the only way to map the likelihood of an exponential family distribution into a confidence is to linearly transform its natural parameters (for the Beta distribution these are $a-1, b-1$) to the positive real line.

(For example, a light that turns on when two heads are flipped, another light turns on when two tails are flipped. So, a single sample might induce a likelihood such that $a$ or $b$ goes up by more than 1.)

To answer your question: do the whole Bayesian inference to arrive at the natural parameters of the induced likelihood ($\Delta a, \Delta b$ in your case), and then scale these to your desired "confidence" if it exceeds it. After scaling, you can combine your likelihood with your prior.

To me, this scaling is meaningful: You're saying that the evidence you have saturates after a certain point. (I'm interested in this subject if anyone has any references.)

$\endgroup$
1
  • $\begingroup$ Yes. I was thinking about scaling myself actually but I wasn't quite sure if one can do that. I mean theoretically. It would quite precisely do what I am aiming for though... $\endgroup$
    – Tom
    Apr 4 '13 at 8:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.