I'm going through the process of writing backpropagation for a neural network. In particular I'm building a MNIST classifier.
I'm wondering if it is better to apply the cost function and backpropagation to the non-one-hot output activation.
The advantage I see is that this will encourage all would-be-non-hot values toward 0, rather than simply being marginally less than the maximum value. Likewise, it would encourage the would-be-hot value towards 1.
An additional one-hot output layer could be applied after training to clamp the values to either 0 or 1.
Andrew Ng's machine learning course wanted me to use one-hot encoding. Do my arguments for using the raw-non-hot encoding output carry any value?