For a project I am currently working on, I'm attempting to implement machine learning for a neural network using backpropagation and gradient descent from scratch.

For much of my implementation, I referenced this tutorial: https://machinelearningmastery.com/implement-backpropagation-algorithm-scratch-python/

My question is to do with calculating the error for the output layer of the network. The tutorial above gives this code for the output layer of the network:

error = (expected - output) * transfer_derivative(output)

Where the transfer derivative is the derivative for the activation function for the output neuron, and (expected - output) gives the difference between the expected output of the network for that neuron, and the actual calculated output. This error is then used as the starting point for backprop.

Now, running my code with this method does work and I've been training for my datasets using it. However, it occurred to me while attempting to implement alternative cost/fitness functions that this doesn't seem to take into account the fitness function for the network. Is this a mistake of my understanding or a limitation of the tutorial? For instance, surely if one were using MSE to calculate the final loss for the network, then the derivative of MSE with respect to a given output neuron should be used here. Instead, it seems to just take the straight difference between the output and target data vectors. Now if I want to use, say, cross entropy loss, this would apparently have no impact on this calculation. Is perhaps the derivative of MSE already incorporated here in some manner that I am not understanding? In that case, how would this be adapted to take into account a cross entropy loss function?

Could somebody tell me what I'm missing here and how to adapt my code (which is very similar to that of the tutorial) to take into account different loss functions such as cross-entropy loss? I have tried to parse some online explanations, but often get lost in the math (I'm a programmer more than I am a mathematician, and would appreciate some "nuts and bolts" with respect to how I should adapt my code).

Thank you.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.