Suppose I want to train a deep neural network to perform classification or regression, but I want to know how confident the prediction will be. How could I achieve this?
My idea is to compute the cross entropy for every training datum, based on its prediction performance in the neural meters above. Then, I would train a second neural network for regression, which would take each datum as input, and it's cross entropy as output (one output node). You would then use both networks in practice -- one for prediction of label / value, and the other for prediction of confidence of the first network. (....But would I then need a third network to predict the confidence of the second network, and so on...?!)
Is this a valid idea? Moreover, is it a standard idea commonly used? If not, what would you suggest?