4
$\begingroup$

I've seen this mentioned numerous times, most recently in motivating the MCMC method and description of the Metropolis-Hastings algorithm. The text (Simulation and the Monte Carlo Method, Second Edition. By R.Y. Rubinstein and D. P. Kroese) says:

An added advantage of MCMC is that it only requires specification of the target pdf up to a (normalization) constant.

Can someone please give some intuition on knowing a probability density function up to a constant and why it is an "advantage" in this context?

$\endgroup$
  • 2
    $\begingroup$ This may help. $\endgroup$ – GeoMatt22 Apr 24 '17 at 23:23
  • 2
    $\begingroup$ GeoMatt22 gave a very good link from this site. I would merely comment that in Bayesian inference with the exception of using conjugate priors the posterior distribution can be computationally difficult and it is the norming constant that causes the difficulty. MCMC makes the calculation easier $\endgroup$ – Michael Chernick Apr 24 '17 at 23:49
  • $\begingroup$ So when we know a posterior distribution is proportional to some function of the parameters as in the Bayesian examples you've referenced, how do you explain what exactly that means about what we know about the posterior distribution? How would you illustrate or visualize what knowledge of what a density is proportional to would look like on a graph? $\endgroup$ – Denis Kelleher Apr 25 '17 at 0:44
  • 4
    $\begingroup$ To "visualize", just have no units on the y axis. "Up to a constant" means you can stretch the graph vertically however you want, so units become meaningless. $\endgroup$ – GeoMatt22 Apr 25 '17 at 5:12
  • $\begingroup$ @GeoMatt22 Excellent, except that units (of measurement) are never meaningless or should be regarded as unclear; they are always probability per unit of the variable. It's the magnitude of a prefactor that floats according to the rest of the data. $\endgroup$ – Nick Cox Feb 5 '18 at 11:09
9
$\begingroup$

As explained in the comments, to know a pdf up to a normalizing constant means that for a random variable $X$ with pdf $f(x)$, we know that $$f(x) \propto g(x) \Rightarrow f(x) = c g(x), $$ where $c = \int g(x) dx$ is unknown. Consider the standard normal distribution which has pdf $$f(x) = \dfrac{1}{\sqrt{2\pi}} e^{\frac{-x^2}{2}}\,. $$

Now suppose we are in a situation, where we can only identify that $$f(x) \propto e^{\frac{-x^2}{2}} \,,$$ and thus $g(x) = e^{\frac{-x^2}{2}}$ and the normalizing constant $c = \frac{1}{\sqrt{2\pi}}$ is not known. If we visualize this and plot $f(x)$ and $g(x)$, we get the following graph

enter image description here

The dotted line is the true pdf unknown to us, and the solid line is the known $g(x)$. The shape of the density remains the same, but the function is stretched or squeezed in the vertical direction depending on the unknown normalizing constant.

In Bayesian models, often we know the posterior only upto a normalizing constant, but we want to learn about different characteristics of the posterior distribution: like mean, mode, or quantile of the distribution. Well, clearly the mode of $g(x)$ corresponds to the mode of $f(x)$, so the normalizing constant is not needed. However, to calculate the mean for the distribution

$$E(X) = \int xf(x) dx = \int cx g(x) = c \int xg(x)dx\,. $$

Even if you can solve $\int xg(x)dx$, $c$ is unknown, and you can't find the mean of $X$. Here is where MCMC can be used since it is able to draw samples from the distribution of $X$, without requiring the normalizing constant $c$.

$\endgroup$
  • $\begingroup$ Is the normalizing constant then always found analytically by solving the integral of cg(x) over x and setting equal to 1 if possible/tractable? $\endgroup$ – Denis Kelleher Apr 25 '17 at 13:03
  • 1
    $\begingroup$ @DenisKelleher Yes, if it's tractable, that's how it is found. $\endgroup$ – Greenparker Apr 25 '17 at 13:22
4
$\begingroup$

Adding to other comments and nice answer by Greenparker, normalizing constant is something that is needed so that probability density function integrates to unity and is a proper distribution.

We do not always need the probability densities to be normalized, e.g. when we are using maximum likelihood estimation, or Bayesian MCMC estimation we only need to know the relative relations between the values. For example, if you want to estimate parameter $p$ of binomial distribution by maximizing it's likelihood, then the only the only thing you need is the binomial density known up to a normalizing constant, i.e. $ p^k(1-p)^{n-k} $. The constant is $ \binom n k$ and since it does not change anything about finding maximum of likelihood function $L(p)$, it is not needed.

You can ask: So what? If we can use it or not, then why not use it always? The problem is that while for the commonly used distributions we know the normalizing constants, then they do not have to be obvious for other distributions. Imagine for example a Bayesian model $f(\theta|x) = c^{-1} \,f(x|\theta) \,f(\theta)$, where $c$ is the normalizing constant and $\theta$ is the parameter of interest. It is easy to multiply the prior by likelihood, but to obtain the normalizing constant you need to integrate $c = \int f(x|\theta) \,f(\theta) \,d\theta$ and this is more complicated (even for discrete distribution, with countably finite support, there are cases where you need to sum huge number of elements and it is computationally intensive problem). If you are dealing with complicated, multivariate distributions then solving the integral may be a problem by itself. Hopefully, we have computational tools like MCMC to deal with Bayesian models in form $f(\theta|x) \propto f(x|\theta) \,f(\theta)$, that enable us to draw samples from such distribution and conduct the estimation without knowing the constant.

So there are cases where the distribution is known up to a constant while the constant itself may not be obvious. Moreover, we are able to do many mathematical operations with the distributions without knowing the constant.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.