I'm a software engineer, and I'm working my way through Stanford professor Andrew Ng's online course on machine learning. I'm able to follow most of the math related to gradient descent for linear regression.
One thing I am not clear about is whether there is a typical (best practice) approach to computing the partial derivative of an arbitrary cost function: Are we supposed to compute this derivative by hand, or is there some software that will do it for us? If we use software, would it be like Mathematica, where we enter a symbolic equation, and the software returns a symbolic derivative that we then implement in code?
For concreteness, in gradient descent for linear regression, the linear coefficient update rule is the following with a partial derivative:
The cost function for linear regression is the following:
And so below is the resulting update rule with the partial derivative expanded out:
Are we always supposed to compute that partial derivative by hand, even for arbitrarily complex cost functions?