1
$\begingroup$

We have two distributions, $P$ and $Q$ such that $P$ is our input distribution and $Q$ is our target distribution. The formulation of $KL = \mathbb{E}_{P}\left[\log\frac{P}{Q}\right]$ allows us to approximate Q by minimising the expectation.

According to that formulation I've created a small experiment and below there are two plots, the first one is showing the distributions of $P$ and $Q$ after the first step of the optimisation, and the second plot again at the end of the optimisation process.

Here's a more detailed explanation of how the plots where generated.

Let $\theta\sim\mathcal{N}(0, 1)$ be the parameters of interest and $Q\sim\text{softmax}(\mathcal{N}(-2.3, 1.45))$ be our target distribution.

Suppose $X\sim\mathcal{N}(2, 1.3)$ is our data and $P = \text{softmax}(\theta^{T}X)$ is our likelihood.

Then to compute the KL $\sum_{i}^{n}P_{i}*\log(\frac{P_{i}}{Q_{i}})$ loss and thus our goal is to minimise $\hat{\theta} = \arg\max_{\theta}KL$.

Running the optimisation process for 1000 steps we have the following loss

enter image description here

My question is the following, it seems like the approximation in the beginning and end of optimisation isn't so good in this toy example, why is that ?

enter image description here

enter image description here

Improve this question
$\endgroup$
16
  • 1
    $\begingroup$ 1) If one of the distributions is the target (which usually means a fixed distribution), why are $p$ and $q$ both changing in your plots? 2) It doesn't make sense that the distributions in the second plot minimize the KL divergence. Possible coding error? 3) Could you edit the question to describe what you did in more detail? $\endgroup$
    – user20160
    Commented Mar 24, 2021 at 14:47
  • $\begingroup$ What are you varying to approximate $Q$?? $\endgroup$
    – whuber ♦
    Commented Mar 24, 2021 at 16:20
  • $\begingroup$ @user20160 Thanks for the pointers, I think I might have had an error, which I've fixed and I've also updated the OP. But I'm still uncertain as to why now the plots have still bad approximations. $\endgroup$
    – Kirk Walla
    Commented Mar 24, 2021 at 16:20
  • 1
    $\begingroup$ @Good I do not disagree. We close ambiguous questions as a favor to everyone: during the period when the original proposer is clarifying the text, it prevents the appearance of conflicting answers due to differing interpretations. Anyone who has worked to craft a good answer only to be told it wasn't answering the "intended" question understands how important this process is. $\endgroup$
    – whuber ♦
    Commented Mar 24, 2021 at 16:51
  • 1
    $\begingroup$ @Kirk I do not follow your last comment, because KL divergence doesn't approximate anything: it merely measures a discrepancy between two distributions. $\endgroup$
    – whuber ♦
    Commented Mar 24, 2021 at 16:52

0

Reset to default