What are soft classes? When reading the help page of tf.nn.sparse_softmax_cross_entropy_with_logits, I came across the term soft classes. I read that soft classes have ground truth labels which are a probability distribution on their one hot encodings. How do you intuitively interpret the information a soft class label provides (the probability distribution)?
 A: Hard classes mean one-hot encoded distributions like:
y = [0, 0, 1, 0, 0]

Having labels like that means that any output distribution will be penalized to force the network output the class 2 (zero-indexed).
In some cases, it's ok for the network to output any of two or more classes, e.g., class 0 or class 2. A particular example is a SkipGram model for word embeddings (if we ignore the issues of a large vocabulary). You can't do that with hard classes, other than create two training instances with two different labels:
x -> [1, 0, 0, 0, 0]
x -> [0, 0, 1, 0, 0]

As a result, the weights will probably bounce back and forth, because the two examples push them in different directions. That's when soft classes can be helpful. They allow you to train the network with the label like:
x -> [0.5, 0, 0.5, 0, 0]

Note that this is a valid probability distribution and it matches the cross-entropy loss. But it's more explicit for the network: classes 0 and 2 are both correct, so the network should boost both. It will learn to match [0.5, 0, 0.5, 0, 0] distribution directly as opposed to alternating [1, 0, 0, 0, 0] and [0, 0, 1, 0, 0].
By the way, if you're interested in more subtleties of cross-entropy losses in tensorflow, read also this question on SO.
