The posterior

$$ P(M|D) = \frac{P(D|M) P(M)}{P(D)} $$

I think in my first class it said that $P(D)$ is a fixed thing, so it is permissible to ignore it for optimization purpose and use

$$ P(M|D) \propto P(D|M) P(M) $$

and I accepted that, it made sense then.

But finally thinking about it, this is effectively saying that $$ P(M|D) \propto P(M,D) $$ which seems just weird. The $P(M|D)$ is a single-variable probability, whereas the $P(M,D)$ is a joint, so not even the dimensionality matches. Is it because we think of this applying only to a single datapoint, so think of $P(M|D)$ and the other factors as numbers rather than as distribution-like functions?


To expain the question further, ignore probability and imagine two generic functions $f(x)$ and $g(x,y)$. If we see an equation $$ f(x) \propto g(x,y) $$ I think it would seem like nonsense, similar to adding my height and weight together to get a single number "86".

However in the probability case, a good comment below reminded that in $$ p(m|d) = \frac{p(m,d)}{p(d)} $$ the fraction on the right is meaningful, although it involves a 2-variable function on the top, and a 1-variable function on the bottom.

It would make sense to me if we could say that in this case, $d$ is a constant, so $p(m,d)$ is really a function of just one variable, and similarly on the left, $p(m|d)$ is a probability distribution with $m$ varying and $d$ fixed. That way there is just one variable, $m$, on both sides. But is that the right way to think? If not, then what?

  • 1
    $\begingroup$ Please refer to stats.stackexchange.com/questions/12112/…. (If I understand your confusion correctly, you should replace "=" signs by proportional signs.) $\endgroup$
    – ocram
    Apr 5, 2018 at 5:36
  • $\begingroup$ Yes, mistake! I typed ~ instead of \sim in the posting source. It is fixed now. Maybe the question can be understood now.? $\endgroup$ Apr 6, 2018 at 4:18
  • $\begingroup$ The referenced other question (12112) does not answer $\endgroup$ Apr 6, 2018 at 4:19
  • $\begingroup$ I just wrote an answer to a similar XV question that should answer your queries about the proportionality sign $\propto$ coded as \propto [not \sim since $\sim$ is used for connection from rv to distribution] and its meaning. $\endgroup$
    – Xi'an
    Apr 6, 2018 at 4:57
  • $\begingroup$ Regarding your edit, you should really refer to some kind of probability handbook $p(m,d)$ is a joint distribution of two variables, same $p(m|d)$ involves two random variables. If $d$ would be constant, then $p(m,d) = p(m)$ and $p(m)/p(d)$ would not have to return result in $[0,1]$, so it wouldn't be a probability. $\endgroup$
    – Tim
    Apr 12, 2018 at 7:06

1 Answer 1


$$ P(D, M) = \underbrace{P(D|M)}_\text{likelihood} \,\underbrace{P(M)}_\text{prior} $$

by the definition of conditional probability, same definition can be applied to obtain

$$ \underbrace{P(M|D)}_\text{posterior} = \frac{P(D, M)}{P(D)} $$

The constant $P(D)$ can be dropped, as described in numerous threads, e.g. here, here, here, here, here, or here, what leads to

$$ P(M|D) \propto P(D, M) = P(D|M)\,P(M) $$

It can be dropped from the computation if you are not interested in estimating the conditional probabilities directly, e.g. when using Naive Bayes algorithm, where you are only interested in finding the highest peak in the probability, or when using MCMC algorithms in Bayesian setting, that can deal with sampling from unnormalized distributions.

When using maximum likelihood approach (see also this thread), while being interested in estimating a parameter, given observed some data $P(M|D)$, we use as a proxy the conditional distribution of observing the data, given the parameter $P(D|M)$ and call this the likelihood function. Maximum likelihood does not use the priors $P(M)$, while Bayesians consider them as important part of the model. So technically yes, you could drop $P(M)$ as well if you don't care about the priors.

  • $\begingroup$ Yes, this is in accord with what I learned in the class. It is not exactly my question, so I see I did not ask it clearly! My question: P(M|D) is a function of one variable. P(M,D) is a function of two variables. How can a function of one variable be proportional to a function of two variables? $\endgroup$ Apr 7, 2018 at 7:40
  • $\begingroup$ @isolatedstudent ask yourself: if $P(M, D)$ is a function of two variables, how $P(M|D) = \frac{P(M, D)}{P(D)}$ could be a function of one variable? It is not a function of one variable, it is a function of a variable conditioned on another variable. $\endgroup$
    – Tim
    Apr 7, 2018 at 9:32
  • $\begingroup$ thank you, that comment is helpful and related to the question. $\endgroup$ Apr 10, 2018 at 7:03
  • $\begingroup$ I think the easy answer is to consider everything as numbers, not as functions. But the issue is still bothering me. $\endgroup$ Apr 11, 2018 at 5:59
  • $\begingroup$ @isolatedstudent conditional probability is a clearly function of two (or more) variables, so I'm afraid that I don't understand what is unclear for you in here. Maybe edit your question to make it more clear what exactly is that you don't understand? Then I may add more details to my answer as well. $\endgroup$
    – Tim
    Apr 11, 2018 at 6:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.