2
$\begingroup$

I quote the whole related documents about Bayesian theory, and I am confused by a specific part "from the Bayesian viewpoint there is only a single data set D (namely the one that is actually observed)". My question is, suppose we have a few choices for values of parameter w, but we only have one observation of data set D, and if data set D is limited and bias, there is possible no co-existence of some possible value of w and D together, when how do we calculate in such case P(D|w)? Thanks.

For example, if we want to calculate P(D = win lottery | w = women) and in observed data set D, there is no data about women, how do we going to do? Treating it as zero posterior (P(w = woman| D = win lottery)) seems not very perfect?

enter image description here enter image description here enter image description here

regards, Lin

$\endgroup$

1 Answer 1

5
$\begingroup$

Using Bayes theorem is not the same as using Bayesian statistics. You are mixing two different things.

If you knew what is the conditional probability of person's gender given his luck in lottery $\Pr(\text{gender} \mid \text{win})$ and the unconditional probability distribution of winning $\Pr(\text{win})$, then you could apply Bayes theorem to compute $\Pr(\text{win} \mid \text{gender})$. Notice that I did not use anywhere here terms such as prior, likelihood, or posterior, since they have nothing to do with such problems. (You could use naive Bayes classifier for such problems, but first is is not Bayesian since it does not use priors, and second you have insufficient data for it.)

As your quote mentioned, in Bayesian approach we have prior, likelihood and posterior. Likelihood is a conditional distribution of data given some parameter. Prior is distribution of this parameter that you assume a priori before seeing the data. Posterior is the estimate given the data you have and your prior.

To give concrete example illustrating it, let's assume that you have data about some coin since you threw it once and observed a head, let's call it $X$. Obviously, $X$ follows Bernoulli distribution parametrized by some parameter $p$ that is unknown and we want to estimate it. We do not know what is $p$, but we have likelihood function $f(X \mid p)$, that is probability mass function of Bernoulli distribution over $X$ parametrized by $p$. To learn about $p$ Bayesian way, we assume prior for $p$. Since we have no clue what $p$ could be, we can decide to use weekly informative "uniform" Beta(1,1) prior. So out model becomes

$$ X \sim \mathrm{Bernoulli}(p) \\ p \sim \mathrm{Beta}(\alpha, \beta) $$

where $\alpha = \beta = 1$ are parameters of beta distribution. Since beta is conjugate prior for Bernoulli distribution, we can easily compute the posterior distribution of $p$

$$ p \sim \mathrm{Beta}(\alpha + 1, \beta) $$

and it's expected value

$$ E(p \mid X) - \frac{\alpha + 1}{\alpha + \beta + 1} = 0.66 $$

so given the data we have and assuming Beta(1,1) prior, expected value of $p$ is $0.66$.

$\endgroup$
17
  • 1
    $\begingroup$ @LinMa $ p(a|b) = \frac{ p(b|a)p(a) }{ \sum p(b|a)p(a) } $ $\endgroup$
    – Tim
    Commented Aug 20, 2016 at 6:15
  • 1
    $\begingroup$ @LinMa as you can see from the formula for Bayes theorem, p(b) is not mentioned anywhere. $\endgroup$
    – Tim
    Commented Aug 20, 2016 at 8:21
  • 1
    $\begingroup$ @LinMa it's $p(b)$ understood as $p(b) = \sum p(b|a)p(a)$. And Beta(1,1) is "uninformative" since it's flat, i.e. a priori you assume any value of $p$ as equally likely. $\endgroup$
    – Tim
    Commented Aug 20, 2016 at 21:46
  • 1
    $\begingroup$ @LinMa in first case it's prior, in second posterior. If it's still unclear maybe you should start with some handbook on Bayesian statists? $\endgroup$
    – Tim
    Commented Aug 21, 2016 at 6:48
  • 1
    $\begingroup$ @LinMa so what is your likelihood and what is your prior? I just gave simple example, you can use any distributions that make sense and likelihood and prior. $\endgroup$
    – Tim
    Commented Aug 22, 2016 at 5:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.