What is the loss function in neural networks?

Question

I'm trying to understand or visualise what a cost function looks like and how exactly we know what it is. The job of an ANN is to learn from labeled data by weight updating, so that we can make good future predictions given new data (using those weights). However, the weights are updated using back-propagation (using gradient descent). How should I mentally view the loss function? The loss means the error of the output for all possible inputs, but where do we actually derive it from?

Answer 1

The backpropagated deltas are derived via the chain rule of calculus. Notice that, although they are valid over all inputs, in the weight update step they are multiplied with the actual activation of that layer.

For the loss function we usually use MSE for linear layers or cross-entropy for softmax layers such that the backpropagated error becomes the difference of the prediction and the target.

I suggest for a detailed understanding to study the topic in the deep learning book by Goodfellow et al.: http://www.deeplearningbook.org/

A more limited treatment of can be found here: Backpropagation with Softmax / Cross Entropy

Answer 2

You ask for simple explanation how neutral network should train.

• The cost function used in a network depends on what you want to do and sometimes the network architecture. For a regression problem, the most common is least-squares. For classification, cross-entropy is popular.
• Imagine you want your network to output 10, but it actually gives you 9. Assuming least square cost function, your error would be 1.
• We need to adjust the network parameters because there is an error. This error (1 in our example) would propagate back in the network by something known as chain rule.
• We repeat and keep training until our error rate is minimized.