1
$\begingroup$

Both MCMC and Legendre quadrature are numerical methods for integration.

Method 1: MCMC $$E[g(X)] = \int f(x) g(x) \, dx$$

Method 2: Gauss-Legendre quadrature $$\int_{0.5}^{1.5} e^x \cos x \, dx$$ $$= \sum_{i=1}^{k} c_i\cdot f(x_i)$$ where $f(x)=e^x \cos x, \text{ with weights } c_i, \text{ and nods } x_i$.

Example: Gauss-Legendre (requested in comments)

Suppose I want to integrate at 3 points on, $f(x) = e^x \cos x$

3 Nods are: $x_i =[0.7746, 0, -0.7746]$
3 Weights are: $w_i = [0.5556, 0.8888888888888888, 0.5556]$

Thus, $$\int_{0.5}^{1.5} e^x \cos x \, dx = \sum_{i=1}^{3} c_i\cdot f(x_i)=1.275$$

You can also refer here: https://math.okstate.edu/people/yqwang/teaching/math4513_fall14/Notes/gaussian.pdf

So, which method to use to integrate out $\theta$?
$$f(x) = \int f(x,\theta) p(\theta) \, d\theta$$

My goal is to calculate the marginal distribution. I am using Gauss-Legendre, but can I also use MCMC? What's the difference? I am a bit confused.

$\endgroup$
7
  • $\begingroup$ I am not familiar with Gauss- Legendre quadrature. So what exactly are the $x_i$s? $\endgroup$ Jun 28, 2016 at 19:01
  • $\begingroup$ @Greenparker: usually these are zeros of the interpolating family. $\endgroup$
    – Alex R.
    Jun 28, 2016 at 19:10
  • $\begingroup$ @AlexR. I thought $c_i$s are the weights? $\endgroup$ Jun 28, 2016 at 19:16
  • $\begingroup$ Hi, I have updated the question with an example of Gauss-Legendre quadrature. $\endgroup$
    – user13985
    Jun 28, 2016 at 19:23
  • $\begingroup$ You shouldn't use the same symbol, the lower-case $x$, to refer both to the random variable and to the variable of integration, in things like $$ \operatorname{E}(g(X)) = \int f(x)g(x)\,dx $$ (where, as you see, I wrote capital $X$ for the random variable and lower-case $x$ for the variable of integration. Without distinguishing those two things, how would you understand something like $F(x) = \Pr(X\le x)$, or the difference between $F_X(3.5)$ and $F_Y(3.5)$, when those would be $F_X(x)$ and $F_Y(x)$ when $x=3.5$? $\qquad$ $\endgroup$ Jun 28, 2016 at 20:54

1 Answer 1

1
$\begingroup$

The difference is that MCMC is stochastic whereas quadrature methods are deterministic. The latter are defined by picking weights and interpolation points, that are usually determined by the zeros of the family of orthogonal polynomials you choose. Quadrature methods are generally superior to simpler numerical integration schemes like the trapezoid rule, in that they give higher accuracy for less computation.

However quadrature methods start to fail very badly in high dimensions because of the vast number of points you need to interpolate at, along with the fact that quadrature methods are blind to large variations in your function. As well quadrature methods do not work well with functions that are difficult to approximate with polynomials. With VEGAS monte carlo methods, you usually try to (randomly) pick areas where your function has larger values, which causes them to be more efficient than quadrature methods in a number of (ideal) situations.

$\endgroup$
11
  • $\begingroup$ For MCMC, can you integrate out variables? I only remember I can get random samples, and then calculate mean and variance. $\endgroup$
    – user13985
    Jun 28, 2016 at 19:25
  • $\begingroup$ WIth your updated question, are you basically asking if MCMC can return what looks to be a function of $x$? $\endgroup$
    – Alex R.
    Jun 28, 2016 at 19:27
  • $\begingroup$ Yes, I am calculating the marginal distribution: $f(x) = \int f(x,\theta) p(\theta) d\theta$ $\endgroup$
    – user13985
    Jun 28, 2016 at 19:30
  • $\begingroup$ Then the answer is likely no, unless you either do repeated sampling for a grid of points $x$, or you make a variational assumption, for example by assuming $f(x,\theta)=g(x)h(\theta)$. The same applies to quadrature methods. $\endgroup$
    – Alex R.
    Jun 28, 2016 at 19:31
  • $\begingroup$ I thought MCMC is a numerical integration method, what I am missing here? And marginal distribution is an integration of the joint distribution. $\endgroup$
    – user13985
    Jun 28, 2016 at 19:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.