3
$\begingroup$

Here's an answer from one of the questions in the newsletter. I'm not trying to suggest that it's incorrect, I just don't understand why it's an acceptable practice. I know that simplifying assumptions are made all the time, but why is this useful?

\begin{equation} \begin{split} p(\theta|y) &= \frac{p(y|\theta)p(\theta)}{p(y)} \\ &\propto\frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)}\theta^{\alpha-1}(1-\theta)^{\beta-1}*\binom{n}{y}\theta^y(1-\theta)^{n-y} \\ &\propto\theta^{\alpha-1}(1-\theta)^{\beta-1}*\theta^y(1-\theta)^{n-y} \\ &\propto\theta^{\alpha+y-1}(1-\theta)^{\beta+n-y-1} \\ &=\frac{\Gamma(\alpha+y-1)\Gamma(\beta+n-y-1)}{\Gamma(\alpha+\beta+n-1)}\theta^{\alpha+y-1}(1-\theta)^{\beta+n-y-1} \end{split} \end{equation}

My issue is that we go from "proportional-to" to "equals", which is fine, but what exactly is this saying? As in, I understand how we go from each line to the line following it, but I don't understand what this tells us. Sure, we eventually get back to a Beta distribution, but only after removing a ton of values. Are these values not important? Does it mean anything profound that a Beta prior with a binomial likelihood has the property of conjugacy? It can make calculations simpler, sure, but is this useful in practice, if so why?

EDIT 1:
So are these constants actually thrown out, or accounted for later? I understand that constants don't matter with respect to proportionality and integration, but throwing them out completely just seems counter-intuitive to me. Sure, if you're multiplying/dividing a bunch of numbers by an expression with a constant, and all you care about is the relationship of those numbers to each other, then you can throw out constants and the relationship remains the same. Is that what's going on here?

EDIT 2:
From what I can tell, we get rid of constants, simplify, and then take the integral and normalize that integral. By virtue of the normalization, we're inherently accounting for the constants that we removed before. Is this correct? I think I understand now, for whatever reason, I kept thinking of the original equation as a scalar value rather than a probability distribution.

$\endgroup$
  • 2
    $\begingroup$ You can ignore all constants of proportionality in any probability density function because at the end you know it has to have unit total area. For instance, it suffices to know the PDF of the standard normal distribution is proportional to $\exp(-x^2/2)$; finding the constant (equal to $1/\sqrt{2\pi}$) is just an exercise in integration. That's all that's going on here. Answers to the rest of your questions are available here by searching on keywords like conjugate prior. $\endgroup$ – whuber Aug 8 '13 at 20:37
  • $\begingroup$ @whuber So are these constants actually thrown out, or accounted for later? I understand that constants don't matter with respect to proportionality and integration, but throwing them out completely just seems counter-intuitive to me. Sure, if you're multiplying/dividing a bunch of numbers by an expression with a constant, and all you care about is the relationship of those numbers to each other, then you can throw out constants and the relationship remains the same. Is that what's going on here? $\endgroup$ – Steve P. Aug 8 '13 at 20:40
  • $\begingroup$ The constants can always be accounted for--just do the integral! Here it's been done for you: where do you think all the gammas came from in the last equation? :-) Another way to read this derivation is to erase all the intermediate lines, leaving an equality for $p$. $\endgroup$ – whuber Aug 8 '13 at 20:44
  • $\begingroup$ @whuber I felt stupid for having asked the question, but since it can help other people, I un-deleted it. If you or anyone else could provide a more detailed explanation, that would be wonderful. I'm pretty sure that I follow what you're saying, but for me, at least, it's a lot more helpful to see a full explanation. $\endgroup$ – Steve P. Aug 8 '13 at 22:15
  • $\begingroup$ @whuber From what I can tell, we get rid of constants, simplify, and then take the integral and normalize that integral. By virtue of the normalization, we're inherently accounting for the constants that we removed before. Is this correct? I think I understand now, for whatever reason, I kept thinking of the original equations as a scalar value rather than a probability distribution. $\endgroup$ – Steve P. Aug 8 '13 at 22:26
5
$\begingroup$

This seems to be a fairly common issue when beginning to work with these problems; I've had similar questions quite a few times in person.

As you suggest in comments, the normalizing constant is implied by the fact that you must be left with a density at the end.

Slightly less informally, at every step of the derivation, we're really dealing with something that integrates to 1, though stating this fact is (as here) usually omitted.

In some simpler cases where this kind dropping of constants is done it may be possible to keep the constants around and do the explicit integral (and the first times you tackle those problems, is worth doing explicitly), but there are times when the required integral is difficult or sometimes impossible to evaluate algebraically, even though we can show that it must have the required value -- in most cases we can't evaluate a term like $p(y)$ except by "the whole expression still integrates to 1, so that term must be..." (though of course, you must remain aware that the thing must be integrable and so on - one can only hand-wave away what can really be hand-waved; in some situations physicists might be able to cancel out infinities but we really can't).

If it would help, you could write "$= c\cdot$" for "$\propto$" at every intermediate line (or rather use $c_1$, $c_2$ etc anywhere the constant changes). It's still an equality at the end and there's only one possible value for the final constant (if you couldn't make the argument that there's only one possible value, you have to ponder whether it might not be the case).

I call this approach 'spot the density'. In fact, being able to spot (multiples of) densities (and probability functions in the discrete case) has given me shortcuts on a host of integrations (/summations) on non-probability problems. Knowing enough densities is like having a table of integrals for free.

What happens if we keep an equality all the way through?

\begin{split} p(\theta|y) &= \frac{p(y|\theta)p(\theta)}{p(y)} \\ &=\frac{1}{p(y)}\frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)}\theta^{\alpha-1}(1-\theta)^{\beta-1} \cdot \binom{n}{y}\theta^y(1-\theta)^{n-y} \\ &=\left[\frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)p(y)}\binom{n}{y}\right]\theta^{\alpha-1}(1-\theta)^{\beta-1}\cdot\theta^y(1-\theta)^{n-y} \\ &=\left[\frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)p(y)}\binom{n}{y}\right]\theta^{\alpha+y-1}(1-\theta)^{\beta+n-y-1} \\ &=\left[\frac{\Gamma(\alpha)\Gamma(\beta)}{\Gamma(\alpha+\beta)p(y)}\binom{n}{y}\frac{\Gamma(\alpha+\beta+n-1)}{\Gamma(\alpha+y-1)\Gamma(\beta+n-y-1)}\right]\\&\quad\quad\cdot\frac{\Gamma(\alpha+y-1)\Gamma(\beta+n-y-1)}{\Gamma(\alpha+\beta+n-1)}\theta^{\alpha+y-1}(1-\theta)^{\beta+n-y-1} \end{split}

We're left to say "what is the term in the brackets $[]$?" Given the data, it's a constant. The value of the constant is immediately clear from integrating both sides w.r.t $\theta$. The value in doing it the way it was done before should be clear.

$\endgroup$
  • $\begingroup$ Thanks for responding. I definitely understand what's going on now. It's been a long time since I've done anything stats related, and tbh I'm pretty sure I learned it completely incorrectly the first time around-- or rather just memorized techniques instead of actually understanding what I was doing. Again, thanks for your help. Hopefully this time, now that I'm much more motivated and interested, I'll learn stats correctly. $\endgroup$ – Steve P. Aug 9 '13 at 0:10
  • $\begingroup$ Yes, while it was clear you already understood, it was a good question which deserved a fuller answer - indeed, I might come back and improve it later. On that deeper understanding and motivation - I know it can take a while. Some decades ago I was into my third year of doing stats subjects (and mostly doing pretty well in them) before I finally said "Oh, now I get it!" $\endgroup$ – Glen_b -Reinstate Monica Aug 9 '13 at 0:17
  • $\begingroup$ Oh, I'm not complaing, I appreciated the additional explanation... I feel that my biggest issue was that I did not put enough effort into math when I was an undergrad. Unfortunately, I was able to get good grades without truly understanding the material. I thought this was awesome at the time (I was pre-med), now that I'm extremely interested in data analysis/machine learning, I feel incredibly stupid for not caring more when I was younger. Nonetheless, I'm studying hard now, and can't wait for that moment that I feel mathematically mature. I hope I get there soon, but it's a long road. $\endgroup$ – Steve P. Aug 9 '13 at 0:26
  • $\begingroup$ I'm still waiting for that feeling. $\endgroup$ – Glen_b -Reinstate Monica Aug 9 '13 at 0:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.