I'm going through the videos in Andrew Ng's free online machine learning course at Stanford. He discusses Gradient Descent as an algorithm to solve linear regression and writing functions in Octave to perform it. Presumably I could rewrite those functions in R, but my question is doesn't the lm() function already give me the output of linear regression? Why would I want to write my own gradient descent function? Is there some advantage or is it purely as a learning exercise? Does lm() do gradient descent?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ I think you may need to implement gradient descent yourself in another language in cases where there is no good solution in R (for instance, some variation of regression, like regularized regression with big data) $\endgroup$– Manoel GaldinoCommented Apr 16, 2012 at 16:32
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
Gradient descent is actually a pretty poor way of solving a linear regression problem. The lm() function in R internally uses a form of QR decomposition, which is considerably more efficient. However, gradient descent is a generally useful technique, and worth introducing in this simple context, so that it's clearer how to apply it in more complex problems. If you want to implement your own version as a learning exercise, it's a worthwhile thing to do, but lm() is a better choice if all you want is a tool to do linear regression.
-
$\begingroup$ @Martin Could lm() still be better if we have many features ? $\endgroup$ Commented Feb 25, 2016 at 11:13
$\begingroup$
$\endgroup$
The reason online gradient is useful is for large scales applications. In any case, now there are libraries that implement it so you don't need to program it. It is a good way to learn how things work.
In Leon Bottou words:
Large-scale machine learning was first approached as an engineering problem. For instance, to leverage a larger training set, we can use a parallel computer to run a known machine learning algorithm or adapt more advanced numerical methods to optimize a known machine learning objective function. Such approaches rely on the appealing assumption that one can decouple the statistical aspects from the computational aspects of the machine learning problem.
This work shows that this assumption is incorrect, and that giving it up leads to considerably more effective learning algorithms. A new theoretical framework takes into account the effect of approximate optimization on learning algorithms.
The analysis shows distinct tradeoffs for the case of small-scale and large-scale learning problems. Small-scale learning problems are subject to the usual approximation–estimation tradeoff. Large-scale learning problems are subject to a qualitatively different tradeoff involving the computational complexity of the underlying optimization algorithms in non-trivial ways. For instance, Stochastic Gradient Descent (SGD) algorithms appear to be mediocre optimization algorithms and yet are shown to perform extremely well on large-scale learning problems.
