What does it mean intuitively to know a pdf "up to a constant"?

Question

I've seen this mentioned numerous times, most recently in motivating the MCMC method and description of the Metropolis-Hastings algorithm. The text (Simulation and the Monte Carlo Method, Second Edition. By R.Y. Rubinstein and D. P. Kroese) says:

An added advantage of MCMC is that it only requires specification of the target pdf up to a (normalization) constant.

Can someone please give some intuition on knowing a probability density function up to a constant and why it is an "advantage" in this context?

Answer 1

As explained in the comments, to know a pdf up to a normalizing constant means that for a random variable $X$ with pdf $f(x)$, we know that $$f(x) \propto g(x) \Rightarrow f(x) = c g(x), $$ where $c = \int g(x) dx$ is unknown. Consider the standard normal distribution which has pdf $$f(x) = \dfrac{1}{\sqrt{2\pi}} e^{\frac{-x^2}{2}}\,. $$

Now suppose we are in a situation, where we can only identify that $$f(x) \propto e^{\frac{-x^2}{2}} \,,$$ and thus $g(x) = e^{\frac{-x^2}{2}}$ and the normalizing constant $c = \frac{1}{\sqrt{2\pi}}$ is not known. If we visualize this and plot $f(x)$ and $g(x)$, we get the following graph

The dotted line is the true pdf unknown to us, and the solid line is the known $g(x)$. The shape of the density remains the same, but the function is stretched or squeezed in the vertical direction depending on the unknown normalizing constant.

In Bayesian models, often we know the posterior only upto a normalizing constant, but we want to learn about different characteristics of the posterior distribution: like mean, mode, or quantile of the distribution. Well, clearly the mode of $g(x)$ corresponds to the mode of $f(x)$, so the normalizing constant is not needed. However, to calculate the mean for the distribution

$$E(X) = \int xf(x) dx = \int cx g(x) = c \int xg(x)dx\,. $$

Even if you can solve $\int xg(x)dx$, $c$ is unknown, and you can't find the mean of $X$. Here is where MCMC can be used since it is able to draw samples from the distribution of $X$, without requiring the normalizing constant $c$.

Answer 2

Adding to other comments and nice answer by Greenparker, normalizing constant is something that is needed so that probability density function integrates to unity and is a proper distribution.

We do not always need the probability densities to be normalized, e.g. when we are using maximum likelihood estimation, or Bayesian MCMC estimation we only need to know the relative relations between the values. For example, if you want to estimate parameter $p$ of binomial distribution by maximizing it's likelihood, then the only the only thing you need is the binomial density known up to a normalizing constant, i.e. $ p^k(1-p)^{n-k} $. The constant is $ \binom n k$ and since it does not change anything about finding maximum of likelihood function $L(p)$, it is not needed.

You can ask: So what? If we can use it or not, then why not use it always? The problem is that while for the commonly used distributions we know the normalizing constants, then they do not have to be obvious for other distributions. Imagine for example a Bayesian model $f(\theta|x) = c^{-1} \,f(x|\theta) \,f(\theta)$, where $c$ is the normalizing constant and $\theta$ is the parameter of interest. It is easy to multiply the prior by likelihood, but to obtain the normalizing constant you need to integrate $c = \int f(x|\theta) \,f(\theta) \,d\theta$ and this is more complicated (even for discrete distribution, with countably finite support, there are cases where you need to sum huge number of elements and it is computationally intensive problem). If you are dealing with complicated, multivariate distributions then solving the integral may be a problem by itself. Hopefully, we have computational tools like MCMC to deal with Bayesian models in form $f(\theta|x) \propto f(x|\theta) \,f(\theta)$, that enable us to draw samples from such distribution and conduct the estimation without knowing the constant.

So there are cases where the distribution is known up to a constant while the constant itself may not be obvious. Moreover, we are able to do many mathematical operations with the distributions without knowing the constant.