0
$\begingroup$

In the conventional approach dynamics of the weights is simply a "slipping" along the hill to the bottom of the valley: $$ \frac{d \mathbf{W}}{dt} = -\nabla_{\mathbf{W}} L $$ In the mean field approach one replaces the dynamics in the space of weights $\mathbf{W} \in \mathbb{R}^n$ by the dynamics in the space of distributions $p(\mathbf{W})$, which turns into the common aproach, by taking $p(\mathbf{W}) \sim \delta(\mathbf{W})$.

Then the output of the predictive is given by: $$ \widehat{f}(\mathbf{W}, x) = \int p(\mathbf{w}, t) \phi(\mathbf{w}, x) d \mathbf{w} $$ And the loss for quadratic loss function is given by: $$ L_{\text{mf}} (p) = \frac{1}{2} \int_{X} \left(\int p(\mathbf{w}, t) \phi(\mathbf{w}, x) d \mathbf{w} - f(x)\right)^2 d \mu(x) $$ Where $f(x)$ is the ground truth, and the $\mu(x)$ on the input space.

Opposed to the case of space of weights,which in general is rather complicated function of weights this loss is quadratic in $p$. From the expression one really can observe that, but is there some intuition, that going to the space of distributions one would get such result?

The resulting expression for the mean field gradient descent looks rather complicated: $$ \frac{\partial p}{\partial t} = -\nabla_{\mathbf{W}} \cdot (p \nabla_{\mathbf{W}} [\nabla_p L_{\text{mf}}]) $$ In the right hand side there is a differential operator of third order acting on the loss function, which does not look interpretable. Is there some intuition why one could expect this result, or just expect that additional derivatives have to emerge?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.