0
$\begingroup$

The posterior

$$ P(M|D) = \frac{P(D|M) P(M)}{P(D)} $$

I think in my first class it said that $P(D)$ is a fixed thing, so it is permissible to ignore it for optimization purpose and use

$$ P(M|D) \propto P(D|M) P(M) $$

and I accepted that, it made sense then.

But finally thinking about it, this is effectively saying that $$ P(M|D) \propto P(M,D) $$ which seems just weird. The $P(M|D)$ is a single-variable probability, whereas the $P(M,D)$ is a joint, so not even the dimensionality matches. Is it because we think of this applying only to a single datapoint, so think of $P(M|D)$ and the other factors as numbers rather than as distribution-like functions?

Edit:

To expain the question further, ignore probability and imagine two generic functions $f(x)$ and $g(x,y)$. If we see an equation $$ f(x) \propto g(x,y) $$ I think it would seem like nonsense, similar to adding my height and weight together to get a single number "86".

However in the probability case, a good comment below reminded that in $$ p(m|d) = \frac{p(m,d)}{p(d)} $$ the fraction on the right is meaningful, although it involves a 2-variable function on the top, and a 1-variable function on the bottom.

It would make sense to me if we could say that in this case, $d$ is a constant, so $p(m,d)$ is really a function of just one variable, and similarly on the left, $p(m|d)$ is a probability distribution with $m$ varying and $d$ fixed. That way there is just one variable, $m$, on both sides. But is that the right way to think? If not, then what?

$\endgroup$
  • 1
    $\begingroup$ Please refer to stats.stackexchange.com/questions/12112/…. (If I understand your confusion correctly, you should replace "=" signs by proportional signs.) $\endgroup$ – ocram Apr 5 '18 at 5:36
  • $\begingroup$ Yes, mistake! I typed ~ instead of \sim in the posting source. It is fixed now. Maybe the question can be understood now.? $\endgroup$ – isolatedstudent Apr 6 '18 at 4:18
  • $\begingroup$ The referenced other question (12112) does not answer $\endgroup$ – isolatedstudent Apr 6 '18 at 4:19
  • $\begingroup$ I just wrote an answer to a similar XV question that should answer your queries about the proportionality sign $\propto$ coded as \propto [not \sim since $\sim$ is used for connection from rv to distribution] and its meaning. $\endgroup$ – Xi'an Apr 6 '18 at 4:57
  • $\begingroup$ Regarding your edit, you should really refer to some kind of probability handbook $p(m,d)$ is a joint distribution of two variables, same $p(m|d)$ involves two random variables. If $d$ would be constant, then $p(m,d) = p(m)$ and $p(m)/p(d)$ would not have to return result in $[0,1]$, so it wouldn't be a probability. $\endgroup$ – Tim Apr 12 '18 at 7:06
1
$\begingroup$

$$ P(D, M) = \underbrace{P(D|M)}_\text{likelihood} \,\underbrace{P(M)}_\text{prior} $$

by the definition of conditional probability, same definition can be applied to obtain

$$ \underbrace{P(M|D)}_\text{posterior} = \frac{P(D, M)}{P(D)} $$

The constant $P(D)$ can be dropped, as described in numerous threads, e.g. here, here, here, here, here, or here, what leads to

$$ P(M|D) \propto P(D, M) = P(D|M)\,P(M) $$

It can be dropped from the computation if you are not interested in estimating the conditional probabilities directly, e.g. when using Naive Bayes algorithm, where you are only interested in finding the highest peak in the probability, or when using MCMC algorithms in Bayesian setting, that can deal with sampling from unnormalized distributions.

When using maximum likelihood approach (see also this thread), while being interested in estimating a parameter, given observed some data $P(M|D)$, we use as a proxy the conditional distribution of observing the data, given the parameter $P(D|M)$ and call this the likelihood function. Maximum likelihood does not use the priors $P(M)$, while Bayesians consider them as important part of the model. So technically yes, you could drop $P(M)$ as well if you don't care about the priors.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Yes, this is in accord with what I learned in the class. It is not exactly my question, so I see I did not ask it clearly! My question: P(M|D) is a function of one variable. P(M,D) is a function of two variables. How can a function of one variable be proportional to a function of two variables? $\endgroup$ – isolatedstudent Apr 7 '18 at 7:40
  • $\begingroup$ @isolatedstudent ask yourself: if $P(M, D)$ is a function of two variables, how $P(M|D) = \frac{P(M, D)}{P(D)}$ could be a function of one variable? It is not a function of one variable, it is a function of a variable conditioned on another variable. $\endgroup$ – Tim Apr 7 '18 at 9:32
  • $\begingroup$ thank you, that comment is helpful and related to the question. $\endgroup$ – isolatedstudent Apr 10 '18 at 7:03
  • $\begingroup$ I think the easy answer is to consider everything as numbers, not as functions. But the issue is still bothering me. $\endgroup$ – isolatedstudent Apr 11 '18 at 5:59
  • $\begingroup$ @isolatedstudent conditional probability is a clearly function of two (or more) variables, so I'm afraid that I don't understand what is unclear for you in here. Maybe edit your question to make it more clear what exactly is that you don't understand? Then I may add more details to my answer as well. $\endgroup$ – Tim Apr 11 '18 at 6:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.