Can OLS be considered an optimization technique?

Question

Can ordinary least squares estimation be considered an optimization technique? If so, how can I explain this?

Note:

From an AI perspective, supervised learning involves finding a hypothesis function $h_\vec{w}(\vec{x})$ that approximates the true nature between predictor variables and the predicted variable. Let some set of functions with the same model representation define the hypothesis space $\mathbb{H}$ (That is we hypothesise the true relationship to be a linear function of inputs or a quadratic function of inputs and so forth). The objective is to find the model $h\in\mathbb{H}$ that optimally maps inputs to outputs. This is done by application of some technique to finds optimal values for the adjustable parameters $\vec{w}$ that defines the function $h_w(\vec{x})$. In AI we call this parameter optimization. A parameter optimization technique/model inducer/learning algorithm would for example be the back propagation algorithm.

OLS is used to find/estimate for $\beta$ parameters that defines the linear regression line that optimally maps predictor variables to output variables. This would be parameter optimization in the scenario above.

Answer 1

Yes, it is. In OLS, you are looking for the linear model that provides the "best" fit to the data. Implementation requires specifying some notion of what you mean by "best". OLS works by defining the "best" model as the one that minimizes a certain measure of model error -- in this case, the sum of the squares of the model residuals. The residuals are the part of the data that aren't explained by the model: OLS seeks to give the best description of the data, by minimizing the "total amount" of unexplained variation in the data.

Formally, any operaton in which you are solving for the minimum or maximum of some function can be interpreted as an optimization.

Answer 2

I would have said that OLS is an optimization problem rather than an optimisation technique, as there are many optimization techniques/algorithms that can be used to solve OLS problems (e.g. analytical solutions for [ridge] regression, IRWLS for logistic regression, scaled conjugate gradients for neural nets, etc.). There is no reason why you can't fit an OLS regression model by simple gradient descent, the reason we generally don't is there are more efficient algorithms.

OLS specifies what you are optimising, but you get the same solution from any suitable optimisation technique.

Answer 3

Given explanatory variable $x_i$ and observed variable $y_i$, and a linear regression model $y = a + b x$, OLS is the minimum of: $$ \sum_i (y_i - b x_i - a)^2 $$ over all $a,b$.

In contrast to this, quantile regression is the minimum of: $$ \sum_i |y_i - b x_i - a| $$ over all $a,b$.

EDIT As a minimum, the parameter estimates for $a$ and $b$ are some kind of optimum. And the calculation method, i.e. solving the normal equations, to actually find the estimates can be considered as technique.

So I vote for yes, OLS is a kind of optimization technique. If someone says "I solved the problem using OLS" I would conclude he used the euclidean norm and a linear solver on the normal equations formulation of the problem. I do not know which solver, though.

So the term "OLS" is more similar to "MLE" than to numerical methods like "Simplex algorithm" or "Conjugate Gradient method".