In Andrew Ng's machine learning course, he introduces linear regression and logistic regression, and shows how to fit the model parameters using gradient descent and Newton's method.

I know gradient descent can be useful in some applications of machine learning (e.g., backpropogation), but in the more general case is there any reason why you wouldn't solve for the parameters in closed form-- i.e., by taking the derivative of the cost function and solving via Calculus?

What is the advantage of using an iterative algorithm like gradient descent over a closed-form solution in general, when one is available?

  • 13
    $\begingroup$ I don't think there is a closed form solution for the MLE of the regression parameters in most glms (e.g. logistic regression). Linear regression with normal errors is one exception. $\endgroup$
    – Macro
    Commented Feb 20, 2012 at 2:35
  • 5
    $\begingroup$ Interesting... Does this mean different stats packages might give different answers for logistic regression depending on, e.g., initial parameter settings, number of iterations, multiple local minima, etc.-- or is there a conventional procedure that all good stats packages will follow? (Though I'm sure any differences, if they do exist, are minute in most cases) $\endgroup$
    – Jeff
    Commented Feb 20, 2012 at 4:45
  • 3
    $\begingroup$ (+1) To your question and your comment, Jeff. GLMs using the canonical link (like logistic regression) benefit from the nice properties of convexity. There can be more than one algorithm to solve such problems, but the basic upshot of this is that (modulo some fairly minor details), well-implemented numerical algorithms will give consistent results between them. $\endgroup$
    – cardinal
    Commented Feb 20, 2012 at 10:58
  • 3
    $\begingroup$ I personally dislike Andrew Ng's course because it has led people into believing that Linear Regression is "machine learning". $\endgroup$
    – Digio
    Commented May 19, 2016 at 14:28
  • 2
    $\begingroup$ Related: Do we need gradient descent to find the coefficients of a linear regression model $\endgroup$ Commented Jan 15, 2017 at 23:46

3 Answers 3


Unless the closed form solution is extremely expensive to compute, it generally is the way to go when it is available. However,

  1. For most nonlinear regression problems there is no closed form solution.

  2. Even in linear regression (one of the few cases where a closed form solution is available), it may be impractical to use the formula. The following example shows one way in which this can happen.

For linear regression on a model of the form $y=X\beta$, where $X$ is a matrix with full column rank, the least squares solution,

$\hat{\beta} = \arg \min \| X \beta -y \|_{2}$

is given by


Now, imagine that $X$ is a very large but sparse matrix. e.g. $X$ might have 100,000 columns and 1,000,000 rows, but only 0.001% of the entries in $X$ are nonzero. There are specialized data structures for storing only the nonzero entries of such sparse matrices.

Also imagine that we're unlucky, and $X^{T}X$ is a fairly dense matrix with a much higher percentage of nonzero entries. Storing a dense 100,000 by 100,000 element $X^{T}X$ matrix would then require $1 \times 10^{10}$ floating point numbers (at 8 bytes per number, this comes to 80 gigabytes.) This would be impractical to store on anything but a supercomputer. Furthermore, the inverse of this matrix (or more commonly a Cholesky factor) would also tend to have mostly nonzero entries.

However, there are iterative methods for solving the least squares problem that require no more storage than $X$, $y$, and $\hat{\beta}$ and never explicitly form the matrix product $X^{T}X$.

In this situation, using an iterative method is much more computationally efficient than using the closed form solution to the least squares problem.

This example might seem absurdly large. However, large sparse least squares problems of this size are routinely solved by iterative methods on desktop computers in seismic tomography research.

  • 5
    $\begingroup$ I should mention that there are also numerical accuracy issues that can make the use of the closed form solution to the least squares problem unadvisable. However, this would require a discussion of ill-conditioning that seems likely to be beyond the current understanding of the original poster. $\endgroup$ Commented Feb 20, 2012 at 3:28
  • 22
    $\begingroup$ please don't hesitate to post an answer because you don't think i will understand it. first-- it won't hurt to provide more information, even if takes me some research in order to grasp it. second-- the stackexchange model assumes that this question and answer will benefit others in the future. in other words, don't dumb down your answer based on how much you think the OP knows, or you'll be doing others a disservice. $\endgroup$
    – Jeff
    Commented Feb 20, 2012 at 4:20
  • 2
    $\begingroup$ @Brian, my feeling is your comment hits closer to the heart of the issue and is a bit at odds with the first sentence in the answer. I don't think any least-squares software (in its right mind) employs the closed-form solution. :) $\endgroup$
    – cardinal
    Commented Feb 20, 2012 at 10:53
  • 6
    $\begingroup$ Cardinal- in practice, it's best to use the QR factorization or SVD to solve small scale least squares problems. I'd argue that a solution using one of these orthogonal factorizations is also a "closed form solution" in comparison to using an iterative technique like LSQR. I didn't delve into this in my answer because it needlessly draws attention away from my main point. $\endgroup$ Commented Feb 22, 2012 at 17:31
  • 2
    $\begingroup$ Ill-conditioning? Textbook closed form solution? I love the smell of squared condition numbers in the morning. Have a big condition number? Why not square it and make it even bigger? Have a not so big condition number? Why not square it and make it big. $\endgroup$ Commented May 15, 2016 at 17:32


For linear regression, it's a one step procedure, so iteration of any kind is not needed.

For logistic regression, the Newton-Raphson iterative approach uses the second partial derivatives of the objective function w.r.t. each coefficient, as well as the first partial derivatives, so it converges much faster than gradient descent, which only uses the first partial derivatives.


There have been several posts on machine learning (ML) and regression. ML is not needed for solving ordinary least squares (OLS), since it involves a one-step matrix sandwiching operation for solving a system of linear equations -- i.e., $\boldsymbol{\beta}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$ . The fact that everything is linear means that only a one-step operation is needed to solve for the coefficients. Logistic regression is based on maximizing the likelihood function $L=\prod_i{p_i}$, which can be solved using Newton-Raphson, or other ML gradient ascent methods, metaheuristics (hill climbing, genetic algorithms, swarm intelligence, ant colony optimization, etc).

Regarding parsimony, use of ML for OLS would be wasteful because iterative learning is inefficient for solving OLS.

Now, back to your real question on derivatives vs. ML approaches to solving gradient-based problems. Specifically, for logistic regression, Newton-Raphson's gradient descent (derivative-based) approach is commonly used. Newton-Raphson requires that you know the objective function and its partial derivatives w.r.t. each parameter (continuous in the limit and differentiable). ML is mostly used when the objective function is too complex ("narly") and you don't know the derivatives. For example, an artificial neural network (ANN) can be used to solve either a function approximation problem or supervised classification problem when the function is not known. In this case, the ANN is the function.

Don't make the mistake of using ML methods to solve a logistic regression problem, just because you can. For logistic, Newton-Raphson is extremely fast and is the appropriate technique for solving the problem. ML is commonly used when you don't know what the function is. (by the way, ANNs are from the field of computational intelligence, and not ML).

  • $\begingroup$ Could you mark a difference between topics covered by computational intelligence and machine learning ? $\endgroup$
    – user305883
    Commented Jul 28, 2020 at 12:29
  • $\begingroup$ This answer seems is a bit misleading : dont you agree that we know the derivatives (partial derivatives) of the cost function (not the func that the ANN is approximating of course...since it is obviously unkown) of an artificial neural network (ANN), since otherwise the backprop algo simply could not be computed! From reading this answer it feels like the author is saying the derivatives of cost func are not known... this is very problematic if wrong answers are allowed?! $\endgroup$
    – SheppLogan
    Commented Nov 14, 2020 at 17:56
  • $\begingroup$ Also the fact that the user does not anymore have a profile is dubious...? For me, spreading wring information, and even worse, wrong info packed in true info is very bad. Of course it might not be on purpose.But I would be happy for someone else to give his opinions. $\endgroup$
    – SheppLogan
    Commented Nov 14, 2020 at 18:01
  • $\begingroup$ Or maybe i m missing/forgetting something but in any case I really think this point should be explicitely detailed $\endgroup$
    – SheppLogan
    Commented Nov 14, 2020 at 18:38

Closed-form solution is OK for full-rank system, but if X.T@X is singular, that is non-invertible, use of it causes computational problems.

Though compared with GD (as numerical solution) linear algebra solution, as closed-form, is faster for small matrices.

Size of matrix also matters. Even if matrix is invertible, it can be too huge & thus too costy to invert it (even after decorrelation, or whitening, with SVD), use of solve-method for square matrices or least-squares-approach (last allows approximation for ununique solutions of underdetermined/overdetermined systems) is preferred to avoid doing manual inverse, that accumulates errors when rounding occurs in algorithm.

Analytic solution (e.g. Newton method for root-finding) using derivative calculations is quicker than numerical calculation of derivatives, thus is preferred over the last, if first exists.

Thus, GD seems to be the last choice, as for me! (as told in the answer above e.g. for backpropagation is affordable) -- no arguments for the opposite view I see, if somebody does have?

As for speed, size & precision aims all together, -- Least-Squares is preferred over GD. And besides, most of optimization problems are solved with Linear Programming, not NLP (that can not be calculated with LinAlg, + signal cannot be returned back after non-linear processing, thus need save initial signal, if need, + works with fixed-point numbers, not floating-point numbers).

Concerning Closed-form solution (with regularization) - it has all conveniences of Linear Algebra - just do it with appropriate methods (numpy's or scipy's solve or least-squares), and use whitening (use param whiten=True in sklearn.decomposition.PCA) and regularization to decrease computational cost and achieve valid approximation (do not approximate noise to blur the system, approximate signal to reveal patterns) with proper resampling methods before computational experiment.

p.s. A comparison between gradient descent and Newton's method; gradient-based-methods; Line-search-in-gradient-and-Newton-directions - directions can differ for Newton-method & GD

p.p.s. The spectral condition number plot for regularization parameter evaluation


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.