3
$\begingroup$

It appears that backpropagation is exclusively used to train neural network models. Why not use it to fit other models. For example - Taylor polynomials:

$$ f(x) = c_0+c_1(x-a)+c_2(x-a)^2...+c_n(x-a)^n $$

We can represent it in a graph and take derivatives backwards the same way as everything is differentiable including the center.

                                 Taylor Graph

Any thoughts? Thanks

$\endgroup$
1
  • $\begingroup$ I can't speak for anyone else, but my colleagues and I do things like this from time to time. The PyTorch library makes it almost trivially easy. (But equivalently, one could say that your Taylor polynomial is just a special type of neural network, just one with an unusual architecture. There isn't really a fundamental difference between neural networks and other types of model, when you get down to it.) $\endgroup$
    – N. Virgo
    Commented Feb 19, 2018 at 7:44

2 Answers 2

5
$\begingroup$

I am interpreting your question as backpropagation just referring to a method of computing derivatives. Some people also interpret it to include the actual gradient descent algorithm using the gradient thus computed. @Sid 's answer focuses just on the gradient descent algorithm, and does not address backpropagation as a method for computing derivatives. My answer does.

Backpropagation is the reverse mode (a.k.a. reverse accumulation, a.k.a. reverse pass) of automatic differentiation (a.k.a algorithmic differentiation) applied to neural networks. The reverse mode of automatic differentiation is commonly applied to compute the gradient of objective function, among other things, in nonlinear optimization. This can support steepest descent (gradient descent), conjugate gradient or Quasi-Newton methods, among others. If followed by forward pass automatic differentiation of the gradient (this is called forward over reverse), it can be used to compute a Hessian or Hessian-vector products for use in a Newton method. The forward mode can be used to compute the Jacobian of constraints for nonlinear optimization.

There are whole books on (reverse mode of) automatic differentiation. Here are a few links: https://en.wikipedia.org/wiki/Automatic_differentiation

https://rufflewind.com/2016-12-30/reverse-mode-automatic-differentiation

Step-by-step example of reverse-mode automatic differentiation

http://www.cs.cmu.edu/~wcohen/10-605/notes/autodiff.pdf

There are also many libraries and software tools to support use of automatic differentiation.

$\endgroup$
3
$\begingroup$

The algorithm underlying back propagation is gradient descent, so I am guessing your question is "Why cant we just use gradient descent on any function/Taylor approximation of a function". There are a few points to note here

1) Gradient descent is in some sense the last resort in optimization problems. If the problem is too large/problem is non-convex and having some untenable form, this might be our only hope.

2) Gradient descent can converge very slowly in many cases..see slide 10-7 and 10-8 in http://ee364a.stanford.edu/lectures/unconstrained.pdf for an explanation and an example. There are modifications of the algorithm which converge several times faster (like Newton's method).

3) Gradient descent does not guarantee that a function converges to a global minimum or even a local minimum. The point can be a global minimum/local minimum/saddle point.

The optimization we do in Neural Networks is sort of a last resort 'Hail Mary'.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.