0
$\begingroup$

I understood backprop: get gradient wrt a parameter (i.e. the partial derivative) using the chain rule.

In the post http://www.offconvex.org/2016/12/20/backprop/ the authors say that the inefficient way to compute gradient would be quadratic in the terms of nodes of the graph?

I didn't understand what they are trying to say..they are trying to get gradients in feedforward manner? How could that be done even if inefficiently.

So, my question is what is this way of computing gradients inefficiently in a neural net?

Are there other similar techniques too? I always heard backprop was the only way.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

With some Algebra you can derive the gradient of each weight parameter independently and use it to get their derivatives in any order you like. It is basically the derivative of the objective function w.r.t. each weight parameter. So, it is possible to compute the derivatives in a feedforward manner.

However, if you compute the derivatives starting from the last layer weights (those close to the output) to the first layers', you can re-use a lot of computations. This is because earlier layers' derivatives require the values of the derivatives computed for the later layers.

If you compute the derivative of each weight without considering the order of computation, then you have to repeat many computations which results in a quadratic time complexity.

So, the order in which you compute the derivatives is crucial.

Improve this answer
$\endgroup$
2
  • $\begingroup$ could you explain using a simple example .. how feedforward gradient is done? you still have to use gradients from upper layers...so it isn't feedforward right? $\endgroup$
    – david nadal
    Jan 6, 2018 at 5:23
  • $\begingroup$ you still have to use the gradients from the upper layers yah. So it isn't feedforward in that sense. The gradients would be computed the same way as for backpropagation. But instead of computing the upper layer gradients once, you will be computing them multiple times if you naively compute the gradients starting from the lower layer weight parameters (those closer to the input layer). $\endgroup$
    – IssamLaradji
    Jan 6, 2018 at 8:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.