I'm going through the process of writing backpropagation for a neural network. In particular I'm building a MNIST classifier.

I'm wondering if it is better to apply the cost function and backpropagation to the non-one-hot output activation.

The advantage I see is that this will encourage all would-be-non-hot values toward 0, rather than simply being marginally less than the maximum value. Likewise, it would encourage the would-be-hot value towards 1.

An additional one-hot output layer could be applied after training to clamp the values to either 0 or 1.

Andrew Ng's machine learning course wanted me to use one-hot encoding. Do my arguments for using the raw-non-hot encoding output carry any value?

  • $\begingroup$ what is meant by "non one hot output"? Like [0.6,0.2,0.2] which is converted to [1,0,0]? $\endgroup$ – user0 Dec 22 '17 at 1:21
  • $\begingroup$ @nan Yes, the first I would call the raw, non-one-hot values, and the second one-hot. Please let me know if there is better terminology. $\endgroup$ – Tom Hale Dec 22 '17 at 11:43

I assume that what ever is backpropagated has to be a probability distribution (the vector sum = 1). I.e., it's already been through a softmax.

It would be interesting to see which works better empirically, but I think the raw output contains more information.

Consider raw output [0.01, 0.45, 0.54]. We then convert this to [0,0,1]. Say we have truth [0,0,1]. (We really want something like [1e-10,1e-10,1-2e-10].)

If we use the one hot [0,0,1], we don't change anything. The error is [0,0,0].

If we use the raw output, however, we end up changing the weights to increase the ouput's certainty (eg, we decrease the first and second and increase the third), with error [0.01,0.45,-0.44].

This might not even be evident if we only evaluated the accuracy or ROC area for a classifier, but perhaps you would see a difference in calibration (the ability to give accurate probability estimates)--I think that the raw output would lead to better calibrated models, because by one hot encoding you are essentially removing information about numerical probabilities.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.