Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
It appears that backpropagation is exclusively used to train neural network models. Why not use it to fit other models. For example - Taylor polynomials:
$$ f(x) = c_0+c_1(x-a)+c_2(x-a)^2...+c_n(x-a)^n $$
We can represent it in a graph and take derivatives backwards the same way as everything is differentiable including the center.
Any thoughts? Thanks
I am interpreting your question as backpropagation just referring to a method of computing derivatives. Some people also interpret it to include the actual gradient descent algorithm using the gradient thus computed. @Sid 's answer focuses just on the gradient descent algorithm, and does not address backpropagation as a method for computing derivatives. My answer does.
Backpropagation is the reverse mode (a.k.a. reverse accumulation, a.k.a. reverse pass) of automatic differentiation (a.k.a algorithmic differentiation) applied to neural networks. The reverse mode of automatic differentiation is commonly applied to compute the gradient of objective function, among other things, in nonlinear optimization. This can support steepest descent (gradient descent), conjugate gradient or Quasi-Newton methods, among others. If followed by forward pass automatic differentiation of the gradient (this is called forward over reverse), it can be used to compute a Hessian or Hessian-vector products for use in a Newton method. The forward mode can be used to compute the Jacobian of constraints for nonlinear optimization.
There are whole books on (reverse mode of) automatic differentiation. Here are a few links: https://en.wikipedia.org/wiki/Automatic_differentiation
https://rufflewind.com/2016-12-30/reverse-mode-automatic-differentiation
Step-by-step example of reverse-mode automatic differentiation
http://www.cs.cmu.edu/~wcohen/10-605/notes/autodiff.pdf
There are also many libraries and software tools to support use of automatic differentiation.
The algorithm underlying back propagation is gradient descent, so I am guessing your question is "Why cant we just use gradient descent on any function/Taylor approximation of a function". There are a few points to note here
1) Gradient descent is in some sense the last resort in optimization problems. If the problem is too large/problem is non-convex and having some untenable form, this might be our only hope.
2) Gradient descent can converge very slowly in many cases..see slide 10-7 and 10-8 in http://ee364a.stanford.edu/lectures/unconstrained.pdf for an explanation and an example. There are modifications of the algorithm which converge several times faster (like Newton's method).
3) Gradient descent does not guarantee that a function converges to a global minimum or even a local minimum. The point can be a global minimum/local minimum/saddle point.
The optimization we do in Neural Networks is sort of a last resort 'Hail Mary'.
