In Bishop's book, he talks about Bayesian Logistic regression and how to compute it. I understood approximating posterior distribution using Laplace approximation. But I don't quite understand predictive distribution part. Specifically:

The predictive distribution for class $C_1$, given a new feature vector $\phi(x)$, is obtained by marginalizing with respect to the posterior distribution $p(w|t)$... $$p(C_1|\phi,t) = \int{p(C_1|\phi,w)p(w|t)dw} \approx \int{\sigma(w^T\phi)q(w)dw} \ \ \cdots (1)$$ To evaluate the predictive distribution, we first note the function $\sigma(w^T\phi)$ depends on $w$ only through its projection onto $\phi$. Denoting $a=w^T\phi$, we have $$\sigma(w^T\phi) = \int{\delta(a-w^T\phi)\sigma(a)da}$$ where $\delta$ is the dirac delta function. From this we obtain $$\int{\sigma(w^T\phi)q(w)dw} =\int{\sigma(a)p(a)da}$$ where $$p(a) = \int{\delta(a-w^T\phi)q(w)dw}$$ ... Thus our variation approximation to the predictive distribution becomes $$p(C_1|t) = \int{\sigma(a)p(a)da} = \int{\sigma(a)N(a|\mu_a, \sigma^2_a)da}\ \ \cdots (2)$$

What exactly is the problem with equation (1) that we need to substitute it with dirac functions? At the end, he gets equation (2) but both (1) and (2) have a sigmoid function and a normal distribution. Why all the hassle to change from (1) to (2)?

I guess I don't quite understand what it means when he says

function $\sigma(w^T\phi)$ depends on $w$ only through its projection onto $\phi$.


(1) involves a multivariate Gaussian with potentially off-diagonal covariance matrix terms. You can see this from the previous section of Bishop, which gives the form of $q(w)$ as $$ q(w) = \mathcal{N}(w \mid w_{\text{MAP}}, S_N), $$ where $S_N$ can generally have off-diagonal terms. This is replaced by an integral over a single variable (as opposed to $D$ variables) with a univariate Gaussian. You see that even then, the integral isn't analytically tractable, which the author fixes by approximating the sigmoid function with a probit, $\Phi(\lambda a)$ and makes use of the identity, $$ \int \Phi(\lambda a) \mathcal{N}(a \mid \mu \sigma^2) da = \Phi\left(\frac{\mu}{(\lambda^{-2} + \sigma^2)^{1/2}}\right), $$ which wouldn't have been possible had he stuck with the original multivariate Gaussian with an unknown covariance. (I'm not sure if an analogous identity exists for a Gaussian with arbitrary covariance, though if you are able to reduce the expression to one that makes use of a simpler identity, then why not?)

As for the projection comment. Consider the following integral, $$ S = \int_{-\infty}^\infty \int_{-\infty}^\infty f(a x + by) g(x,y) dx dy $$ where $a$ and $b$ are constants. The integral is over the entire two-dimensional plane. Imagine, on the plane, the directional vector $(a,b)'$. You can define a line spanning the space along this direction. The inner product of any vector with $(a,b)'$ is a projection of that vector on this line. For example, suppose we have a vector $(c,d)'$ that's orthogonal to $(a,b)'$, so that $(c,d) (a,b)' = 0.$ Then $(x+c, y+d) (a,b)' = (x,y)(a,b)'.$ This means that the value of $f(ax + by)$ is determined entirely by the projection of $(x,y)'$ on the line defined by the directional vector $(a,b)'.$ Any change in a direction orthogonal to that line has no impact on the value of $f$.

We can then perceive this integral as an integral of $f$ over the line parameterized by a single variable $\lambda,$ given by a weighting function over that parameter, where the weight is given by $$p(\lambda) = \int \delta(\lambda - (a x + b y)) g(x,y) dx dy.$$ Consider what this integral is, for a given value of $\lambda.$ This is an integral of $g$ only over the line defined by $a x + b y = \lambda,$ which is perpendicular to the vector $(a,b)'.$ If $g$ is a two-dimensional probability distribution then you can think of this as marginalizing the distribution $g(x,y)$ over the direction orthogonal to $(a,b)'.$ If $g$ is a Gaussian, this marginalization is easy to do, and is done by Bishop in the passage you skipped over.

$S$ is an integral over $x$ and $y$, though you can re-imagine it as an integral over the line given by the direction $(a,b)'$, where at each incremental step, another integral is taken over the line orthogonal to $(a,b)'$ at that given point (so a rotation in the directions in which we integrate). This can be generalized in $D$ dimensions to an integral over a one-dimensional line, where at each increment a $D-1$ dimensional integral is taking place. Any function that depends only on a point's projection on the the line is a constant of the $D-1$ dimensional integral.

  • $\begingroup$ What do you mean by off-diagonal? It's kind of odd that Bishop never mentioned that $S_n$ can be off-diagonal(or it's likely I missed it). After some thinking, I thought the reason behind using Direc delta was that $\sigma(w^T\phi)$ was in fact scalar (as opposed to a sigmoid function in equation (2)) and needed to be converted into sigmoid function in order to do write it as a convolution of sigmoid and gaussian. $\endgroup$ – MoneyBall Apr 20 '17 at 17:09
  • $\begingroup$ Just look at the form of $S_N$ in 4.143. In general it can have off-diagonal terms. The only way it's completely diagonal is if the transformed feature space is completely uncorrelated (which could be the case if, say, it was the result of a PCA transformation). Also I don't understand what you mean. $\sigma(x)$ is a scalar function no matter what its argument is. $\endgroup$ – Bridgeburners Apr 20 '17 at 17:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.