0
$\begingroup$

I'm reading through Regularizing Neural Networks By Penalizing Confident Output Distributions and I'm stuck on the equation in section 3.2. It's not clear to me at all that the self-entropy of the model output distribution $H(p_\theta(\mathbb{y}_i,\mathbb{x}))$, is the same as $D_{KL}(u||p_\theta(\mathbb{y}_i,\mathbb{x}))$.

I did try to reason through it as follows:

We know the penalised loss is:

$$ \begin{align} \mathcal{L}(\theta) & = - \sum_{i}\log p_{\theta}(\mathbb{y}_i|\mathbb{x}) - \beta H(p_{\theta}(\mathbb{y_i}|\mathbb{x})) \\ & = - \sum_{j=i}\log p_{\theta}(\mathbb{y}_i|\mathbb{x}) - \beta \sum_{i}p_{\theta}(\mathbb{y_i}|\mathbb{x})\log(p_{\theta}(\mathbb{y_i}|\mathbb{x})) \end{align} $$

and if we only care about minimizing the loss, we can feel free to add a constant term to it, like $\sum_{i}p_{\theta}(\mathbb{y_i}|\mathbb{x})\log(u)$, where $u = 1/K$ (I believe this is constant, because the $u$ comes out of the sum then the sum just adds to 1). It doesn't mean the RHS is the same as before. It just differs by a constant. So by doing this we get some new loss which is just off by a constant:

$$ \begin{align} \mathcal{L}'(\theta) & = - \sum_{j=i}\log p_{\theta}(\mathbb{y}_i|\mathbb{x}) - \beta \sum_{i}(p_{\theta}(\mathbb{y_i}|\mathbb{x})\log(p_{\theta}(\mathbb{y_i}|\mathbb{x})) + p_{\theta}(\mathbb{y_i}|\mathbb{x})\log(u)) \\ & = - \sum_{j=i}\log p_{\theta}(\mathbb{y_i}|\mathbb{x}) - \beta \sum_{i}(p_{\theta}(\mathbb{y_i}|\mathbb{x})\log(p_{\theta}(\mathbb{y_i}|\mathbb{x})/u)) \\ & = - \sum_{j=i}\log p_{\theta}(\mathbb{y_i}|\mathbb{x}) - \beta D_{KL}(p_{\theta}(\mathbb{y_i}|\mathbb{x})||u) \end{align} $$

Presuming I'm on the right track, I don't know a few things:

  • Where did the $\beta$ go?
  • Why are the $u$ and the $p_\theta$ swapped around?
  • Why are they then allowed to "reverse the direction of the KL divergence" to further their argument?
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.