I am following a tutorial from Analytics Vidhya on creating a neural network to recognize handwritten digits (the classic example).

The code from the tutorial states "First we need to define the cost of the neural network, and then optimize with backpropagation. However it seems like this is not what is happening in the code. It appears that the cross entropy is being minimized first, then it is sent to the backprop algorithm. Can someone explain the process or reasoning of First minimizing the cross entropy and then using a backpropagation algorithm?

Confused because I read this article from stackoverflow about the softmax_cross_entropy_with_logits function. This gave my the impression that


already minimizes the cross entropy?


Is this just two separate ways of optimizing the weights of the network? It seems as if he is just defining the cost so that the backprop algorithm can minimize it, even though softmax_cross_entropy already minimizes something?

Here are the lines of code I am uncertain about (where tf is tensorflow, and y the labels):

output_layer = tf.matmul(hidden_layer, weights['output']) + biases['output']

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(output_layer, y))

optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

I understand that there is a difference in the backprop algorithm compared to optimizing cross entropy. But what I am confused about is why they are both occurring here.

Posting here because I saw another post on this community regarding softmax and cross entropy. Apologies if this is the incorrect location.


1 Answer 1


There is no such thing as "minimizing the cross entropy and using a backpropagation algorithm (BPA)". BPA's objective is to find weights that minimize the loss function, which is cross entropy here. The confusion arises from tensorflow (TF)'s way of doing things. It's like building up a circuitry, connecting cables between pieces and prepare them to work together when required.

For example, below line binds cost variable to the output of tf.reduce_mean, whose input is bound to the output of tf.nn.softmax... and so on. It's just preparation. There is no minimization at this line of code. We're instructing the network to "calculate cross entropy with last layer's and real outputs, take the mean, and equate it to the variable (tensor) cost, while running".

cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(output_layer, y))

After that, we choose our optimizer and call minimize, which still doesn't start minimizing. It's our instruction.

optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

The real calculation starts when the session (tf.Session) is run. If you look into the codes in SO carefully, you'll see sess.run(...) or s.run(...) etc.

  • $\begingroup$ Perfect! Yes I suppose it is the way TensorFlow is doing things which confused me initially. With this being said, doesn't this technically only reduce the weights in the output layer then? When would the hidden layers weights be modified by an optimization algorithm? Or does the optimizer follow the input all the way back to the hidden layer weights? $\endgroup$
    – ghDev
    May 26, 2019 at 23:39
  • $\begingroup$ yes, the optimizer follows all the way back. $\endgroup$
    – gunes
    May 27, 2019 at 5:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.