Linear subspace property of Gaussian integrals In this talk at 19:32, the following method of reducing a multivariate integral is discussed:
Let $\theta \in \mathbb{R}^p$ be the parameter vector, $\mathbf{x}_i \in \mathbb{R}^p$ be the $i$-th data vector, $y_i \in \{-1,1\}$, be the corresponding binary observation. Also let $\phi(\cdot)$ denote the logistic sigmoid function. We want to simplify the following integral (product of sigmoid and multivariate Gaussian terms)
\begin{align*}
z & = \int_{\mathbb{R}^p} \phi(y_i \mathbf{x}_i^T \theta)~ \mathcal{N}(\theta|\mu, \Sigma) d \theta \\
\end{align*}
This can be expressed as an expectation with respect to $\theta$:
$$
\mathbb{E}_{\theta} [\phi(y_i \mathbf{x}_i^T \theta)]
$$
and since $u := y_i \mathbf{x}_i^T \theta$ is a sclar, we can write this as:
$$
\mathbb{E}_{u} [\phi(u)]
$$
which is a one-dimensional expectation. I understand that $u$ is Gaussian, but I do not understand why the distribution the expectation is taken with respect to can be changed like that
 A: I think the answer goes along these lines: Essentially it is Fubini and Substitution for multivariate integrals.
Let us assume that $d=2, \Sigma=\text{Id}$ and $\mu = 0$. Notice that the density takes the shape
$$\mathcal{N}(\Theta, \text{Id}, 0) = \text{const.}~ e^{-|\Theta|^2/2}$$
in this case. I will ignore the constant to keep the formulae simple. Take a rotation matrix
$$ A = \begin{pmatrix} \cos(\alpha) & -\sin(\alpha) \\ \sin(\alpha) & \cos(\alpha) \end{pmatrix}$$
such that $y_i A^T x_i = (c,0)^T$ where $c \neq 0$. If we execute the substitution (see https://en.wikipedia.org/wiki/Integration_by_substitution#Substitution_for_multiple_variables) $\Theta \mapsto A\Theta$ then we get that
$$\int_{\mathbb{R}^2} \phi(y_ix_i^T\Theta) \mathcal{N}(\Theta, \text{Id}, 0) d\Theta = \int_{\mathbb{R}^2} \phi(y_ix_i^TA\Theta) e^{-|A\Theta|^2/2} d\Theta$$
but notice that a vector $\Theta$ and its rotated variant $A\Theta$ have precisely the same length, i.e. $|A\Theta| = |\Theta|$ so that this simplifies to
$$\int_{\mathbb{R}^2} \phi((y_iAx_i)^T\Theta) e^{-|\Theta|^2/2} d\Theta = \int_{\mathbb{R}^2} \phi(c\Theta_1) e^{-\Theta_1^2/2} e^{-\Theta_2^2/2} d\Theta$$
At this point we use the Theorem of Fubini (https://en.wikipedia.org/wiki/Fubini%27s_theorem) which allows us to write the integral over the total domain $\mathbb{R}^2$ of $\Theta$ as an iterated integral
$$\int_{\mathbb{R}} \int_{\mathbb{R}} \phi(c\Theta_1) e^{-\Theta_1^2/2} e^{-\Theta_2^2/2} d\Theta_2 d\Theta_1 = \left( \int_{\mathbb{R}} e^{-\Theta_2^2/2} d\Theta_2 \right ) \cdot \left( \int_{\mathbb{R}} \phi(c\Theta_1) e^{-\Theta_1^2/2}  d\Theta_1\right)$$
Now if we do not forget about the normalizing constant above and drag it along all the time then we have $(2\pi)^{-1}$ in front which we write as $\sqrt{2\pi}^{-1} \sqrt{2\pi}^{-1}$. The expression becomes
$$\left( \sqrt{2\pi}^{-1} \int_{\mathbb{R}} e^{-\Theta_2^2/2} d\Theta_2 \right ) \cdot \left( \sqrt{2\pi}^{-1} \int_{\mathbb{R}} \phi(c\Theta_1) e^{-\Theta_1^2/2}  d\Theta_1\right)$$
The first integral vanishes (because it integrates to the constant $\sqrt{2\pi}$) and the second integral is what you wanted: an integral over the logistic function "in a univariate Gaussian expectation". If the amount of dimensions is more than $2$ then everything still works but gets a little messier. If $\Sigma$ is not the identity matrix and/or $\mu \neq 0$ then $\Sigma$ can be diagonalized unitarily (basic linear algebra) so you use some substitution $\Theta \mapsto U\Theta - \mu$ or so in advance in order to force the situation $\Sigma=\text{Id}$ and $\mu=0$ just as you do in the case of a univariate normal.
