Note: there is a nearly identical question on Stack Overflow. However, I seem to be missing something... or maybe it's just that Python isn't my first language ;)

For a neural network library, I've implemented a number of activation and cost functions along with their derivatives. Any combination of activation and cost function can be chosen for the network's output layer.

For backpropagation, the output layer's error gradient with respect to each node's input is simply the product of the activation derivative and cost derivative (forgive the pseudocode):

$gradient_j = activation.derivative(y_j) * cost.derivative(y_j, t_j)$

where $y$ is the network's final (activated) output and $t$ is the target output.

How can I implement my Softmax derivative to work here? So far, all activation gradients have been computable using only the $y$ value: logistic, tanh, etc. Softmax appears to be a special case; can someone help me out?

  • $\begingroup$ The other site is appropriate for this question and this site is not. Maybe you can get the answer you want by following the post you linked us to. $\endgroup$ – Michael R. Chernick Apr 12 '17 at 0:04
  • $\begingroup$ @Michael Chernick - this site has far more posts re: machine learning than Stack Overflow. S.O. is good for coding help with specific languages; I'm looking for a mathematical explanation for my problem. The link I provided has Python help, but not much math. $\endgroup$ – Hundley Apr 12 '17 at 0:42
  • $\begingroup$ Yes but not on software. Why do you think the other post is on SO? $\endgroup$ – Michael R. Chernick Apr 12 '17 at 1:15
  • $\begingroup$ @hundley $\LaTeX$ might help, although I suspect there are plenty of people on here that will be able to answer your question as is $\endgroup$ – Taylor Apr 12 '17 at 4:10
  • $\begingroup$ @Taylor I gave a LaTeX a shot, but unfortunately I'm not a guru when it comes to proper notation - hence the pseudocode. I think this is what gave Michael the impression I was looking for coding help. Hopefully somebody has an idea on how to solve this... $\endgroup$ – Hundley Apr 12 '17 at 4:32

I can assure you one thing after reading this blog you will have all your questions answered just read it religiously.This blog has the whole implementation from scratch in python just using numpy. http://neuralnetworksanddeeplearning.com

| cite | improve this answer | |
  • $\begingroup$ When you answer a question you need to put eat to the bones. Don't rely completely on a link for the answer. $\endgroup$ – Michael R. Chernick Aug 21 '19 at 21:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.