2
$\begingroup$

This is possibly a silly conceptual question, ... but anyway:

Imagine I have a function:

$f = F(\mathbf{x}) = F(x_1,x_2) = ax_1^2 + bx_2^3,$

where $x_1,x_2 \sim N(0,1)$ for example.

For a naive uncertainty propagation I can MC (Monte Carlo) sample $x_1$ and $x_2$, and find:

hist($f$) = $p(f)$ (unnormalized),

by passing my samples through $F(\mathbf{x})$ and placing each subsequent output onto a histogram. This histogram is 1D, because $F(\mathbf{x})$ has a scalar output.

If I want to do a Laplace approximation on this histogram / uncertainty distribution, I would require a Hessian but I'm at a loss as to where this Hessian should come from. Should it:

  1. Come directly from an equation for the histogram (which I don't have since I only have a frequency distribution) OR
  2. Be built on $F(\mathbf{x})$. With this approach the Hessian would be 2x2 because $F$ is a function of two variables. However as stated before my histogram output is only 1D, so my Laplace approximation should only have a scalar s.d. to work with

Hence I'm not sure which method (or maybe none of these?) is the correct way for calculating the Hessian for the Laplace approximation in this problem. Each way seems to be difficult in some respect.

$\endgroup$
2
$\begingroup$

In your context, the Laplace approximation represents some PDF as a Gaussian.

Therefore the Hessian of interest is for the density, rather than the domain:

To approximate the marginal PDF of $F$, the domain is $F$ (i.e. $\mathbb{R}$) and the range is $p[F]$, i.e. your histogram.

To approximate the joint PDF of $(x_1,x_2)$, the domain is $\mathbb{R}^2$ and the range is $p[x_1,x_2]$ (this is already a Gaussian).

To approximate the joint PDF $p[x_1,x_2,F]$, the domain would be $\mathbb{R}^3$.

The Hessian of $F[x_1,x_2]$ is not exactly relevant to any of these.

I am not certain if the PDF of $F$ for your example is analytically computable. However, for demonstration purposes, if you had instead $F=x_1^2+x_2^2$, then the (marginal) PDF of $F$ would be a chi-squared distribution with 2 degrees of freedom. In this case, to do a Laplace approximation you would compute the Hessian for "your histogram" ... which would then have an analytical form.

$\endgroup$
  • $\begingroup$ Thanks for that answer, it was great! Really insightful. Just two comments: 1. Could you clarify the term "marginal" in this context. I always thought of it as something like $p(x) = \int p(x|\alpha)p(\alpha) d\alpha$, so I'm not 100% sure how it turns up here. 2. Just to clarify your conclusion on this problem. IF I do not have an analytical equation for my histogram, it is not possible to apply a Laplace approximation for said problem? $\endgroup$ – pche8701 Aug 19 '16 at 15:29
  • $\begingroup$ The marginal PDF of $F$ means you just care about frequency of $F$ values, independent of any other variables. Your integral is correct, but conceptually simpler if you think of the integrand as $p(x,\alpha)$. So you are integrating the joint PDF over all values of $\alpha$, which is the "other variable" in this case. For the second point, I believe a Laplace approximation would commonly be used with an "analytical equation". However the equation does not have to be an expicit formula, and is commonly an integral. $\endgroup$ – GeoMatt22 Aug 19 '16 at 15:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.