I am following a tutorial from Analytics Vidhya on creating a neural network to recognize handwritten digits (the classic example).
The code from the tutorial states "First we need to define the cost of the neural network, and then optimize with backpropagation. However it seems like this is not what is happening in the code. It appears that the cross entropy is being minimized first, then it is sent to the backprop algorithm. Can someone explain the process or reasoning of First minimizing the cross entropy and then using a backpropagation algorithm?
Confused because I read this article from stackoverflow about the softmax_cross_entropy_with_logits function. This gave my the impression that
already minimizes the cross entropy?
Is this just two separate ways of optimizing the weights of the network? It seems as if he is just defining the cost so that the backprop algorithm can minimize it, even though softmax_cross_entropy already minimizes something?
Here are the lines of code I am uncertain about (where tf is tensorflow, and y the labels):
output_layer = tf.matmul(hidden_layer, weights['output']) + biases['output'] cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(output_layer, y)) optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
I understand that there is a difference in the backprop algorithm compared to optimizing cross entropy. But what I am confused about is why they are both occurring here.
Posting here because I saw another post on this community regarding softmax and cross entropy. Apologies if this is the incorrect location.