I built a simple classification model that is required to predict a probability distribution for a set of two available classes.
The target distributions are not necessarily delta distributions. i.e the target could have soft labels.
As a sanity check I wanted to test if my model could optimize over a single sample.
I performed two tests. In the first one the target distribution was a delta dist (
[0, 1]), and the second one wasn't (
In both cases the model could achieve a low loss (for the criterion of KL divergence) but the optimization processes and results were very different:
As you can see there are multiple differences between the two:
- The final results are different by multiple orders of magnitude.
- The time of convergence
- One is monotonic and one has ripples
What are the reasons for difference between the optimization processes over a delta and non-delta distributions?
I received different optimizations processes using different optimizers such as SGD, Adam and it's variations.
Here is the code (in pytorch) for a minimal reproducible example:
import torch import torch.nn as nn import torch.nn.functional as F from torch import optim import matplotlib.pyplot as plt torch.set_default_dtype(torch.double) num_features = 10 num_classes = 2 x = torch.rand(num_features).unsqueeze(0) y1 = torch.tensor([0.0, 1.0]) # delta distribution y2 = torch.tensor([0.3, 0.7]) # non-delta distribution model = nn.Sequential( nn.Linear(num_features, num_classes) ) criterion = nn.KLDivLoss() optimizer = optim.Adam(model.parameters(), lr=0.1) epochs = 300 plt.figure() for target, dist_name in zip([y1, y2], ['delta', 'non-delta']): losses =  for epoch in range(epochs): model.zero_grad() scores = model(x) log_probs = F.log_softmax(scores, dim=-1) loss = criterion(log_probs, target) loss.backward() optimizer.step() losses.append(loss) plt.plot(losses, label=dist_name) plt.xlabel('epoch') plt.ylabel('loss') plt.yscale('log') plt.legend() plt.show()