1
$\begingroup$

When reading the paper of Efficient piecewise training of deep structured models for semantic segmentation, I am confused about the derivation in CRF training (section 6). In specific, I do not know how to get the gradient of partition function Z. What's the definition of Z, and how to derive its gradient as shown in the following images.

enter image description here

enter image description here

$\endgroup$
1
$\begingroup$

$Z(\bf{x}; {\bf \theta})$ is the conditional partition function: $$Z({\bf x}; {\bf \theta})=\sum_{{\bf y}\in\mathcal{Y}} \exp\left[-E({\bf y}, {\bf x};{\bf \theta})\right]$$

It is simply the proportionality constant (conditioned on ${\bf x}$) to ensure that the total probability over all ${\bf y}$ correctly adds up to $1$: $$P({\bf y}|{\bf x};{\bf \theta}) \propto \exp\left[-E({\bf y}, {\bf x};{\bf \theta})\right]$$ $$P({\bf y}|{\bf x};{\bf \theta}) = \frac{\exp\left[-E({\bf y}, {\bf x};{\bf \theta})\right]}{Z({\bf x}; {\bf \theta})}$$ Computing the gradient of $\log Z$ uses the chain rule and the derivatives for $\log f(x)$ (i.e. $\frac{1}{f(x)}f'(x)$), $\exp$, and addition. Since we end up summing over $\mathcal{Y}$ twice, we use $\bf y$ and $\bf y'$ to distinguish the two:

$$ \nabla_{\bf \theta} \log Z({\bf x}; {\bf \theta}) = \nabla_{\bf \theta} \log \sum_{{\bf y}\in\mathcal{Y}} \exp\left[-E({\bf y}, {\bf x};{\bf \theta})\right] \\ = \frac{1}{\sum_{{\bf y'}\in\mathcal{Y}} \exp\left[-E({\bf y}, {\bf x};{\bf \theta})\right]}\nabla_{\bf \theta} \sum_{{\bf y}\in\mathcal{Y}} \exp\left[-E({\bf y}, {\bf x};{\bf \theta})\right] \\ = \frac{1}{\sum_{{\bf y'}\in\mathcal{Y}} \exp\left[-E({\bf y}, {\bf x};{\bf \theta})\right]}\sum_{{\bf y}\in\mathcal{Y}} \exp\left[-E({\bf y}, {\bf x};{\bf \theta})\right] \nabla \left[-E({\bf y}, {\bf x};{\bf \theta})\right] \\ = \sum_{{\bf y}\in\mathcal{Y}} \frac{\exp\left[-E({\bf y}, {\bf x};{\bf \theta})\right]}{\sum_{{\bf y'}\in\mathcal{Y}} \exp\left[-E({\bf y}, {\bf x};{\bf \theta})\right]} \nabla \left[-E({\bf y}, {\bf x};{\bf \theta})\right]$$

Here is an example of a paper that uses the term conditional partition function (albeit in the continuous setting).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.