With TensorFlow I've successfully built a neural network that recognises MNIST numbers at 99.2% accuracy. The last layer of the network is the softmax layer, which does output percentages. The AdamOptimizer is used for training and cross entropy is used as the error function.
However, I'm not quite satisfied with the results. I took a closer look at the raw images. The first image (MNIST #449) looks like neither 5 nor 3 and was labeled 5, and the network outputs 3 with 67% confidence and 5 with 32% confidence. This seems acceptable for me, but another image (MNIST #1112) does look like 4, but the network outputted 4 with 6.62% confidence and 8 with 93.36% confidence.
Is there a better approach to make the network output probabilities? For example, maybe make the network detect which pictures are biased, and assign them with a higher entropy?