I built a simple classification model that is required to predict a probability distribution for a set of two available classes.

The target distributions are not necessarily delta distributions. i.e the target could have soft labels.

As a sanity check I wanted to test if my model could optimize over a single sample.

I performed two tests. In the first one the target distribution was a delta dist ([0, 1]), and the second one wasn't ([0.3, 0.7]).

In both cases the model could achieve a low loss (for the criterion of KL divergence) but the optimization processes and results were very different:

Adam optimiztion on two targets

As you can see there are multiple differences between the two:

  1. The final results are different by multiple orders of magnitude.
  2. The time of convergence
    1. One is monotonic and one has ripples


What are the reasons for difference between the optimization processes over a delta and non-delta distributions?


I received different optimizations processes using different optimizers such as SGD, Adam and it's variations.


Here is the code (in pytorch) for a minimal reproducible example:

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch import optim
import matplotlib.pyplot as plt


num_features = 10
num_classes = 2

x = torch.rand(num_features).unsqueeze(0)
y1 = torch.tensor([0.0, 1.0])  # delta distribution
y2 = torch.tensor([0.3, 0.7])  # non-delta distribution

model = nn.Sequential(
    nn.Linear(num_features, num_classes)
criterion = nn.KLDivLoss()
optimizer = optim.Adam(model.parameters(), lr=0.1)
epochs = 300


for target, dist_name in zip([y1, y2], ['delta', 'non-delta']):
    losses = []

    for epoch in range(epochs):

        scores = model(x)
        log_probs = F.log_softmax(scores, dim=-1)
        loss = criterion(log_probs, target)


    plt.plot(losses, label=dist_name)

  • $\begingroup$ use SGD and a learning rate of 0.01 $\endgroup$ – shimao Sep 19 '19 at 23:23
  • $\begingroup$ That indeed eliminates the ripples, but the achieved loss for the non-delta distribution is still lower by multiple orders of magnitude. Why can't the optimizer reach similar loss on the delta distribution? $\endgroup$ – Gal Avineri Sep 20 '19 at 7:13
  • $\begingroup$ I believe I found the answer to this. In the case that the target distribution is a delta function, we would like the output of the softmax to be [0, 1]. In order to reach that output the scores of the linear layer is required to have a difference of infinity. In addition, in order to achieve the same order of magnitude of loss that the non-delta case achieves, the linear model still has to supply huge differences between the scores. This requires a weight matrix with huge weights and it couldn't be achieved with iterations with a small learning rate. $\endgroup$ – Gal Avineri Sep 20 '19 at 8:51
  • $\begingroup$ If you believe you have the answer, you can write an Answer in the box below your post. $\endgroup$ – Sycorax Sep 20 '19 at 14:46

Let's start by denoting the scores produced by the linear layer with $s_1, s_2$, and the probabilities produced by the softmax by $q_1, q_2 = q_1, 1-q_1$.

By the definition of the softmax function we get that $q_1 = {e^{s_1} \over {e^{s_1} + e^{s_2}}}$.

In order to produce a prediction of $[k, 1-k], k \in [0,1]$, the difference between $s_2$ and $s_1$ should be $s_2 - s_1 = log({1 \over k} - 1)$.

This functions looks like this: scores difference vs k

So initially we can see that the optimization goals for the two target distributions are different.

When the target is the delta distribution ($k = 1$) the goal of the optimization is to get to the difference as high as possible. On the other hand when the target is the non delta distribution ($k = 0.7$) the goal is to get a the difference as precise as possible.

I will focus on each case separately.

1. Optimizing over the delta distribution

The issue is that the model doesn't reach a low enough loss. (when comparing to the loss achieved on the non-delta distribution), which means the optimization process can't reach a good enough proximity to the distribution $[1, 0]$.

In order to receive a better proximity, we would require to reach a higher difference $s_2 - s_1$.

If my calculation is correct, when using gradient decent, with each upadate to the parameters the difference $\Delta s = s_2 - s_1$ will be updated as follows $$\Delta s_{t+1} = \Delta s_t -2*\alpha * \|x\|_2^2 * (1 - q_{1,t})$$ where $\alpha$ is the learning rate.

The term $(1 - q_{1,t})$ is bound in the range $[0, 1]$ and $x$ was randomized with small magnitude, so it's norm $\|x\|_2^2$ will also be small.

This means that $\Delta s$ is indeed decreasing over time, but the speed is determined by the magnitude of the learning rate. A small learning rate will result in a slow speed and a bigger learning rate will result in a larger speed.

We can test that using a higher learning rate can achieve better results in that case. learning rate effect on delta distribution.

We can see that indeed higher learning rates achieve lower loss.

2. Optimizing over the non-delta distribution

The issue in the optimization process is that the optimization process is oscillating, and also it seems that it is slower compared to the speed we achieved on the other distribution.

I mentioned that the optimization over this distribution is the process of reaching a precise $\Delta s$. This might indicate that the oscillations in the loss can be caused by oscillating over a local minimum.

We can try to reduce the learning: lr effect on non-delta

It seems that lower learning rate reduced the oscillation, but the optimization speed is still much slower compared to what was achieved on the other distribution.

Looking at the original oscillations of the loss w.r.t the non-delta distribution, I noticed that the loss goes up for a few successive iterations, despite having a low learning rate.

I than researched on how the Adam algorithm is defined. Using the default of $\beta_1 = 0.9$ we approximately use the past 10 gradients in order to update the parameters. I than hypothesized that it might be caused due to using too long of a history.
I.e if the direction of the gradient has changed because we leaped over the minimum, that could cause us to keep going in the same direction for too long.

Therefore using a shorter history of gradients might resolve this.

Here is a test with smaller $\beta_1$ values: different betas on non-delta target

We can see that indeed lower $\beta_1$ causes less oscillation, using the original learning rate.

The elimination of the oscillation also increased the speed of the optimization significantly.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.