Suppose I have the following architecture

enter image description here

Where $ HA_i$ and $OA_i$ are the activated values of the hidden and output nodes respectively, and $W_i$ are the weights between nodes.

I want to find the derivative of the error $e$ w.r.t. $W_1$.

Then my chain rule is the following:

$$ \frac{\partial e}{\partial W_1} = \left( \frac{\partial e}{\partial OA_1} \frac{\partial OA_1}{\partial O_1} \frac{\partial O_1}{\partial HA_1} \right) \frac{\partial HA_1}{\partial H_1} \frac{\partial H_1}{\partial W_1} + \left( \frac{\partial e}{\partial OA_2} \frac{\partial OA_2}{\partial O_2} \frac{\partial O_2}{\partial HA_1} \right) \frac{\partial HA_1}{\partial H_1} \frac{\partial H_1}{\partial W_1} $$

Is this correct?

  • $\begingroup$ While one can guess, you need to define your error function or else it cannot be derived. $\endgroup$ – Björn Lindqvist Oct 22 '18 at 13:32
  • $\begingroup$ @BjörnLindqvist Are you talking about - for example in the case of Softmax where the output layer is fully connected - then there are a few more derivatives? I have gotten smarter now - and my question above would apply to all activation functions (i.e.: MSE, MAD, etc) that do not depend on other output nodes. $\endgroup$ – Edv Beq Jan 12 at 15:25
  • $\begingroup$ You are asking whether your derivative is correct. Your question cannot be answered unless one knows what function you are deriving. You ask "find the derivative of the error" but you first need to answer "What is the error?" $\endgroup$ – Björn Lindqvist Jan 13 at 16:46
  • $\begingroup$ @BjörnLindqvist Well we know that error $e$ is some function $e = f(OA_1, OA_2)$; therefore, $\frac{\partial e} {\partial OA_i} = \frac{\partial f(OA_1, OA_2)} {\partial OA_i}$. Are you saying we are not allowed to write derivatives symbolically? $\endgroup$ – Edv Beq Jan 13 at 17:12
  • $\begingroup$ By the way - by definition a cost function has to have a derivative and be expressed as a sum - and a set of other conditions - so I do not see the issue. I am not saying that I can take any function - use it as a cost function - and expect results. $\endgroup$ – Edv Beq Jan 13 at 17:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.