In this talk at 19:32, the following method of reducing a multivariate integral is discussed:

Let $\theta \in \mathbb{R}^p$ be the parameter vector, $\mathbf{x}_i \in \mathbb{R}^p$ be the $i$-th data vector, $y_i \in \{-1,1\}$, be the corresponding binary observation. Also let $\phi(\cdot)$ denote the logistic sigmoid function. We want to simplify the following integral (product of sigmoid and multivariate Gaussian terms)

\begin{align*} z & = \int_{\mathbb{R}^p} \phi(y_i \mathbf{x}_i^T \theta)~ \mathcal{N}(\theta|\mu, \Sigma) d \theta \\ \end{align*}

This can be expressed as an expectation with respect to $\theta$:

$$ \mathbb{E}_{\theta} [\phi(y_i \mathbf{x}_i^T \theta)] $$

and since $u := y_i \mathbf{x}_i^T \theta$ is a sclar, we can write this as:

$$ \mathbb{E}_{u} [\phi(u)] $$

which is a one-dimensional expectation. I understand that $u$ is Gaussian, but I do not understand why the distribution the expectation is taken with respect to can be changed like that

  • 1
    $\begingroup$ Nothing was changed. $y_i\mathbf{x}_i^\prime \theta$ is simply a scaled version of the marginal of $\theta$ in the $y_i\mathbf{x}_i$ direction. It's a defining property of Normal distributions that all marginals themselves have (univariate) Normal distributions. $\endgroup$
    – whuber
    Commented Jan 23, 2018 at 16:46

1 Answer 1


I think the answer goes along these lines: Essentially it is Fubini and Substitution for multivariate integrals.

Let us assume that $d=2, \Sigma=\text{Id}$ and $\mu = 0$. Notice that the density takes the shape

$$\mathcal{N}(\Theta, \text{Id}, 0) = \text{const.}~ e^{-|\Theta|^2/2}$$

in this case. I will ignore the constant to keep the formulae simple. Take a rotation matrix

$$ A = \begin{pmatrix} \cos(\alpha) & -\sin(\alpha) \\ \sin(\alpha) & \cos(\alpha) \end{pmatrix}$$

such that $y_i A^T x_i = (c,0)^T$ where $c \neq 0$. If we execute the substitution (see https://en.wikipedia.org/wiki/Integration_by_substitution#Substitution_for_multiple_variables) $\Theta \mapsto A\Theta$ then we get that

$$\int_{\mathbb{R}^2} \phi(y_ix_i^T\Theta) \mathcal{N}(\Theta, \text{Id}, 0) d\Theta = \int_{\mathbb{R}^2} \phi(y_ix_i^TA\Theta) e^{-|A\Theta|^2/2} d\Theta$$

but notice that a vector $\Theta$ and its rotated variant $A\Theta$ have precisely the same length, i.e. $|A\Theta| = |\Theta|$ so that this simplifies to

$$\int_{\mathbb{R}^2} \phi((y_iAx_i)^T\Theta) e^{-|\Theta|^2/2} d\Theta = \int_{\mathbb{R}^2} \phi(c\Theta_1) e^{-\Theta_1^2/2} e^{-\Theta_2^2/2} d\Theta$$

At this point we use the Theorem of Fubini (https://en.wikipedia.org/wiki/Fubini%27s_theorem) which allows us to write the integral over the total domain $\mathbb{R}^2$ of $\Theta$ as an iterated integral

$$\int_{\mathbb{R}} \int_{\mathbb{R}} \phi(c\Theta_1) e^{-\Theta_1^2/2} e^{-\Theta_2^2/2} d\Theta_2 d\Theta_1 = \left( \int_{\mathbb{R}} e^{-\Theta_2^2/2} d\Theta_2 \right ) \cdot \left( \int_{\mathbb{R}} \phi(c\Theta_1) e^{-\Theta_1^2/2} d\Theta_1\right)$$

Now if we do not forget about the normalizing constant above and drag it along all the time then we have $(2\pi)^{-1}$ in front which we write as $\sqrt{2\pi}^{-1} \sqrt{2\pi}^{-1}$. The expression becomes

$$\left( \sqrt{2\pi}^{-1} \int_{\mathbb{R}} e^{-\Theta_2^2/2} d\Theta_2 \right ) \cdot \left( \sqrt{2\pi}^{-1} \int_{\mathbb{R}} \phi(c\Theta_1) e^{-\Theta_1^2/2} d\Theta_1\right)$$

The first integral vanishes (because it integrates to the constant $\sqrt{2\pi}$) and the second integral is what you wanted: an integral over the logistic function "in a univariate Gaussian expectation". If the amount of dimensions is more than $2$ then everything still works but gets a little messier. If $\Sigma$ is not the identity matrix and/or $\mu \neq 0$ then $\Sigma$ can be diagonalized unitarily (basic linear algebra) so you use some substitution $\Theta \mapsto U\Theta - \mu$ or so in advance in order to force the situation $\Sigma=\text{Id}$ and $\mu=0$ just as you do in the case of a univariate normal.

  • $\begingroup$ could you explain why you chose $A$ in that way? One of the main points of the question was to understand what distribution I should take the expectation with respect to. Under your approach we should have the expectation wrt marginal distribution of $\theta_1$ $\endgroup$ Commented Jan 24, 2018 at 2:05
  • $\begingroup$ @dimebucker91: 1) I choose $A$ in that way because this allows me to "plit" the sigmoidal term into things that only have to do with $\Theta_1$ and $\Theta_2$ respectively in order to split the whole integral. This is in fact what the lecturer means by 'logistic functions on one axis': The logistic part is not really multivariate but depends only on a one dimensional thing and by rotating the situation in that way we make it explicitly 'visible'. 2) That is not a contradiction, in fact, it's the same thing: marginals of multivariate Gaussians are univariate Gaussians (see $\endgroup$ Commented Jan 24, 2018 at 7:33
  • $\begingroup$ en.wikipedia.org/wiki/…) $\endgroup$ Commented Jan 24, 2018 at 7:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.