Gradient descent vs lm() function in R? I'm going through the videos in Andrew Ng's free online machine learning course at Stanford.  He discusses Gradient Descent as an algorithm to solve linear regression and writing functions in Octave to perform it.  Presumably I could rewrite those functions in R, but my question is doesn't the lm() function already give me the output of linear regression?  Why would I want to write my own gradient descent function?  Is there some advantage or is it purely as a learning exercise?  Does lm() do gradient descent?
 A: Gradient descent is actually a pretty poor way of solving a linear regression problem. The lm() function in R internally uses a form of QR decomposition, which is considerably more efficient. However, gradient descent is a generally useful technique, and worth introducing in this simple context, so that it's clearer how to apply it in more complex problems. If you want to implement your own version as a learning exercise, it's a worthwhile thing to do, but lm() is a better choice if all you want is a tool to do linear regression.
A: The reason online gradient is useful is for large scales applications. In any case, now there are libraries that implement it so you don't need to program it. It is a good way to learn how things work.
In Leon Bottou words: 

Large-scale machine learning was first approached as an engineering
  problem. For instance, to leverage a larger training set, we can use a
  parallel computer to run a known machine learning algorithm or adapt
  more advanced numerical methods to optimize a known machine learning
  objective function. Such approaches rely on the appealing assumption
  that one can decouple the statistical aspects from the computational
  aspects of the machine learning problem.
This work shows that this assumption is incorrect, and that giving it
  up leads to considerably more effective learning algorithms. A new
  theoretical framework takes into account the effect of approximate
  optimization on learning algorithms.
The analysis shows distinct tradeoffs for the case of small-scale and
  large-scale learning problems. Small-scale learning problems are
  subject to the usual approximation–estimation tradeoff. Large-scale
  learning problems are subject to a qualitatively different tradeoff
  involving the computational complexity of the underlying optimization
  algorithms in non-trivial ways. For instance, Stochastic Gradient
  Descent (SGD) algorithms appear to be mediocre optimization algorithms
  and yet are shown to perform extremely well on large-scale learning
  problems.

Large scale learning
sgd project
