2
$\begingroup$

I am currently trying to understand a paper on bayesian neural networks whereby the authors use a bayes by backprop approach to learn weight uncertainties in the neural networks.

I am trying to understand the derivation for proposition 1 in the paper. Particularly, I am not sure how

$$\frac{\partial}{\partial\theta}\int f(\boldsymbol{w},\theta)q(\epsilon)d\epsilon = E_{q(\epsilon)}[\frac{\partial f(w,\theta)}{\partial w} \frac{\partial w}{\partial \theta} + \frac{\partial f(w,\theta)}{\partial \theta}]$$

I am not sure why there is an additional $\frac{\partial f(w,\theta)}{\partial \theta}$ inside the expectation ? since I thought $\frac{\partial f(w,\theta)}{\partial w} \frac{\partial w}{\partial \theta} = \frac{\partial f(w,\theta)}{\partial w}$

$\endgroup$
1
  • $\begingroup$ isn't the first term inside the expectation quantity on the left hand side equal to the right hand side term by chain rule? $\endgroup$
    – calveeen
    Dec 7, 2020 at 1:36

1 Answer 1

3
$\begingroup$

Okay I think this is why there is an additional term. $f$ is a function of $\textbf{w}$ and $\theta$. Since $\textbf{w}$ is composed of $\theta$, we apply chain rule to obtain $\frac{\partial f}{\partial w}\frac{\partial w}{\partial \theta}$. Since $f$ is also a function of $\theta$. we also differentiate with respect to $\theta$, yielding $\frac{\partial f}{\partial \theta}$

$\endgroup$
1
  • $\begingroup$ yes (+1), it's the multivariable chain rule. And the reason why you can swap the order of integration and differentiation is the proposition on slide 7 here $\endgroup$
    – Taylor
    Dec 7, 2020 at 19:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.