1
$\begingroup$

When we estimate the posterior density function, we have the following equation:

$p(x|data) = \frac{P(data|x)*p(x)}{p(data)}$

Let us think that our prior is a continuous distribution, say normal. We are trying to get probability of success in a trial that is Binomial, that is p(x|data). Now, What exactly this multiplication means?

$P(data|x)*p(x)$ ?

P(data|x) - is 1 value, say 0.10, or 0.5, etc. On the other hand p(x) is the value of the continuous probability density function (normal here). How can we multiply the value of pdf with P(data|x)?

P(data|x) is the concrete realization of the binomial trial with some given parameter x. According to Bayes rule p(x) should be the probability of the parameter, but p(x) is density! Without integration it has no sense. Could someone explain please?

$\endgroup$
2
  • 1
    $\begingroup$ You haven't applied Bayes' theorem when you merely multiply those two quantities. You also seem to be very confused about discrete variables--although the values might be discrete, the probabilities of those values, as always, may be any real number between $0$ and $1$. $\endgroup$
    – whuber
    Nov 13, 2016 at 2:21
  • $\begingroup$ I fixed the question - i incorrectly formulated my thought (no descrete). I know what Bayes rule is - im just giving the part of the equation i dont understand - i dont understand how one can multiply P(data|x) by pdf in the numerator. Pdf in this case means sth like const*e^-(z^2)/2. P(data|x) assumes that we calculate binomial as x^k * (1-x)^n-k, where we have x fixed! But then we need to have p(x) - but first, its a distribution, second - prior pdf value at point x is not even probability. So yes, i'd like to understand what stated expression means :) $\endgroup$ Nov 13, 2016 at 2:44

1 Answer 1

2
$\begingroup$

$p$ is a function that returns the density evaluated at its argument; so $p(x)$ is a number -- the value of the prior density evaluated at $x$.

Moving to a more typical statistics notation, where parameters are Greek letters and random variables/observations are Roman letters (presently handwaving the distinction between the latter two):

$p(\theta \mid y) = \frac{p(y \mid \theta) \cdot p(\theta)}{p(y)}$

For any given value of $\theta$, we can find $p(y|\theta)$ (the likelihood) and $p(\theta)$, and thereby take the product of those two numbers, obtaining the numerator and hence (up to the scaling factor in the denominator) the posterior evaluated at $\theta$.

$\endgroup$
13
  • 1
    $\begingroup$ Likelihood is not a probability either. Firstly the likelihood function (regarded as a function of $\theta$) doesn't integrate to 1, and secondly, even if you were regarding it as pdf for the sample (rather than a function of $\theta$), it would more typically be another density. You might perhaps argue that you are multiplying density by density to obtain something proportional to another density. (But I'd say that you could as well regard it as in effect multiplying two numbers to get another number.)... ctd $\endgroup$
    – Glen_b
    Nov 13, 2016 at 5:07
  • 1
    $\begingroup$ ctd ... but even if one of the terms actually were a probability, there's no rule that says you can't multiply a probability by some number that isn't a probability. All manner of valid calculations have the form of scaling a probability or scaling a density. $\endgroup$
    – Glen_b
    Nov 13, 2016 at 5:09
  • 1
    $\begingroup$ The likelihood (regarded as a function of $\theta$) is not a density, but it is evaluated by computing one; it all depends on what you are looking at the first term as representing. However, note that $p(y,\theta) = p(y|\theta) p(\theta)$ so if you want to, you can regard the whole numerator as a single density -- a density in both $y$ and $\theta$. [Of course if you write the conditioning the other way $p(y,\theta) = p(\theta|y) p(y)$, and equate the two ways of writing the joint, then on dividing both sides by $p(y)$, we have Bayes theorem.] $\endgroup$
    – Glen_b
    Nov 13, 2016 at 5:14
  • 1
    $\begingroup$ I just outlined the entire proof in my previous comment. It's not quite clear to me what the difficulty is. Why would there by a problem with scaling a probability by the value of a density or a density by a probability? $\endgroup$
    – Glen_b
    Nov 13, 2016 at 5:23
  • 1
    $\begingroup$ Yes, the joint distribution of $y$ and $\theta$. What's $p(x|y)$? why, it's $p(x,y)/p(y)$. If these sorts of calculations (such as multiplying or dividing densities and/or probabilities) bother you in some way, you ought to have run into the issue already. $\endgroup$
    – Glen_b
    Nov 13, 2016 at 5:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.