# What does a “closed-form solution” mean?

I have come across the term "closed-form solution" quite often. What does a closed-form solution mean? How does one determine if a close-form solution exists for a given problem? Searching online, I found some information, but nothing in the context of developing a statistical or probabilistic model / solution.

I understand regression very well, so if any one can explain the concept with reference to regression or model-fitting, it will be easy to consume. :)

• This question seems to have been something of a magnet for low quality answers for some time; I thought perhaps it should be protected for now. – Glen_b Nov 24 '14 at 15:34

## 5 Answers

"An equation is said to be a closed-form solution if it solves a given problem in terms of functions and mathematical operations from a given generally accepted set. For example, an infinite sum would generally not be considered closed-form. However, the choice of what to call closed-form and what not is rather arbitrary since a new "closed-form" function could simply be defined in terms of the infinite sum." --Wolfram Alpha

and

"In mathematics, an expression is said to be a closed-form expression if it can be expressed analytically in terms of a finite number of certain "well-known" functions. Typically, these well-known functions are defined to be elementary functions—constants, one variable x, elementary operations of arithmetic (+ − × ÷), nth roots, exponent and logarithm (which thus also include trigonometric functions and inverse trigonometric functions). Often problems are said to be tractable if they can be solved in terms of a closed-form expression." -- Wikipedia

An example of a closed form solution in linear regression would be the least square equation

$$\hat\beta=(X^TX)^{-1}X^Ty$$

• Considering that all regression scenarios can be cast as a problem of solving a system of equations, when would there not be a closed-form solution? An ill-posed or sparse problem will require an approximate solution, so is that the case where a closed-form solution does not exist? How about when one uses conjugate gradient descent with regularization? – arjsgh21 Sep 25 '13 at 18:44
• I found this discussion helpful - "Solving for regression parameters in closed-form vs gradient descent" link – arjsgh21 Sep 25 '13 at 18:53
• @arjsgh21 do you still need further clarification on what it means to be a closed form solution? Because your new question seems to be about when are there closed form solutions (or not) in regression problems which is an entirely new topic and should be asked as a new question, in my opinion. – user25658 Sep 25 '13 at 18:57
• It confuses me why CrossValidated is the only "stackexchange forum" that consistently supports obfuscating-but-correct answers over answers which provide understanding. The best answer of the current crop is @Luca's, and is unappreciated. True, it only provides a link, but a great link that is easy to understand. This overly erudite answer only helps solve the problem for folks who already know the answer. :( – Mike Williamson Dec 18 '16 at 22:14
• @MikeWilliamson CrossValidated is the Reddit of StackExchange, anything goes. – NoName Jan 13 '20 at 20:34

Most estimation procedures involve finding parameters that minimize (or maximize) some objective function. For example, with OLS, we minimize the sum of squared residuals. With Maximum Likelihood Estimation, we maximize the log-likelihood function. The difference is trivial: minimization can be converted to maximization by using the negative of the objective function.

Sometimes this problem can be solved algebraically, producing a closed-form solution. With OLS, you solve the system of first order conditions and get the familiar formula (though you still probably need a computer to evaluate the answer). In other cases, this is not mathematically possible and you need to search for parameter values using a computer. In this case, the computer and the algorithm play a bigger role. Nonlinear Least Squares is one example. You don't get an explicit formula; all you get is a recipe that you need to computer to implement. The recipe might be start with an initial guess of what the parameters might be and how they might vary. You then try various combinations of parameters and see which one gives you the lowest/highest objective function value. This is the brute force approach and takes a long time. For example, with 5 parameters with 10 possible values each you need to try $10^5$ combinations, and that merely puts you in the neighborhood of the right answer if you're lucky. This approach is called grid search.

Or you might start with a guess, and refine that guess in some direction until the improvements in the objective function is less than some value. These are usually called gradient methods (though there are others that do not use the gradient to pick in which direction to go in, like genetic algorithms and simulated annealing). Some problems like this guarantee that you find the right answer quickly (quadratic objective functions). Others give no such guarantee. You might worry that you've gotten stuck at a local, rather than a global, optimum, so you try a range of initial guesses. You might find that wildly different parameters give you the same value of the objective function, so you don't know which set to pick.

Here's a nice way to get the intuition. Suppose you had a simple exponential regression model where the only regressor is the intercept: \begin{equation} E[y]=\exp\{\alpha\} \end{equation}

The objective function is \begin{equation} Q_N(\alpha)=-\frac{1}{2N} \sum_i^N \left( y_i - \exp\{\alpha\} \right)^2 \end{equation}

With this simple problem, both approaches are feasible. The closed-form solution that you get by taking the derivative is $\alpha^* = \ln \bar y$. You can also verify that anything else gives you a higher value of the objective function by plugging in $\ln (\bar y + k)$ instead. If you had some regressors, the analytical solution goes out the window.

• Did you implicitly equate "analytical" with "closed-form" in the last sentence? – whuber Sep 24 '13 at 3:35
• I thought then synonymous:mathworld.wolfram.com/Analytic.html – Dimitriy V. Masterov Sep 24 '13 at 4:31
• Did you see the disambiguation comments at the end of that MathWorld page? The issue is that in the present context "analytic" can reasonably be understood in several distinct ways. Also, "analytical" and "analytic" do not mean exactly the same thing (just like "historic" and "historical" have different meanings). – whuber Sep 24 '13 at 13:11
• In many mathematical contexts "analytic" is a precise term of art applied to any function locally expressible as a power series with positive radius of convergence, whereas "analytical" much more broadly is related to decomposability into basic parts. As BabakP's quotations indicate, "closed form" acquires meaning only within some context of generally accepted procedures for combining values (usually assumed to consist of elementary but not transcendental functions). – whuber Sep 24 '13 at 17:57
• @unicorn That would be the case, I think. – Dimitriy V. Masterov Aug 18 '20 at 3:23

I think that this website provides a simple intuition, an excerpt of which is:

A closed-form solution (or closed form expression) is any formula that can be evaluated in a finite number of standard operations. ... A numerical solution is any approximation that can be evaluated in a finite number of standard operations. Closed form solutions and numerical solutions are similar in that they both can be evaluated with a finite number of standard operations. They differ in that a closed-form solution is exact whereas a numerical solution is only approximate.

• While only providing a link, this is definitely the most helpful answer. – Mike Williamson Dec 18 '16 at 22:18
• Wayne's inclusion of a quote from the link quite definitely improved the answer. – Glen_b Jul 7 '17 at 2:24
• Moreover, Luca's link is now dead. – Naramsim Dec 17 '17 at 12:06

Looking for lay terms or the painful verbiage that rigorously defines the meaning? I'll presume lay terms as the other can be found everywhere. Let's say you wanted the closed form solution of the square root of 8. The closed form solution is 2 * (2)^1/2 or two times the square root of two. This is in contrast to the non-closed form solution 2.8284. (see wikipedia square root of 2 to see than at 69 decimal places it is accurate to within 1/10,000) One is absolutely defined in mathematical terms whereas the other is not. A closed form solution provides an exact answer and one that is not closed form is an approximation, but you can get a non closed form solution as close as to a closed form solution as you want. Sounds counter intuitive, but if you need it more accurate, then just grind out a little bit more computations.

• This is an unusual use of the term "closed form." Could you provide a reference? – whuber Sep 24 '13 at 3:34
• Not sure that I can sufficiently provide the level of supporting documentation to win a debate on this without more work than I'm willing to put forth, but here goes. Look on Wikipedia for Closed Form Expression. In the last two sections it describes how closed form solutions are not necessarily required because numerical computation can usually be successfully used to arrive at a solution and the following section that describes how some mathematical programs attempt to generate closed form solutions from numeric values. Closed form solutions are precise (out of space) – Cheesepipe Sep 24 '13 at 3:46
• Wikipedia is fine as a reference. In this case it appears you may have conflated "closed form expression" with "closed form number." They don't mean the same things. – whuber Sep 24 '13 at 3:48

Closed-form = closed (functional) form

Closed means nothing more can go inside; that is, no alternative => only one solution => only one function that can establish the relationship between the outcome and the predictors.

• This is also an unusual use of the term. Could you provide some examples of it being used in this context? I'm mostly surprised because one often hears closed-form/no closed form regarding integrals, which don't really have an outcome or predictors. – Matt Krause Jun 8 '14 at 20:40