Why does linear regression use a cost function based on the vertical distance between the hypothesis and the input data point?

Question

Let’s say we have the input (predictor) and output (response) data points A, B, C, D, E and we want to fit a line through the points. This is a simple problem to illustrate the question, but can be extended to higher dimensions as well.

Problem Statement

The current best fit or hypothesis is represented by the black line above. The blue arrow ($\color{blue}\rightarrow$) represents the vertical distance between the data point and the current best fit, by drawing a vertical line from the point till it intersects the line.

The green arrow ($\color{green}\rightarrow$) is drawn such that it is perpendicular to the current hypothesis at the point of intersection, and thus represents the least distance between the data point and the current hypothesis. For points A and B, a line drawn such that it is vertical to the current best guess and is similar to a line which is vertical to the x axis. For these two points, the blue and the green lines overlap, but they don’t for points C, D and E.

The least squares principle defines the cost function for linear regression by drawing a vertical line through the data points (A, B, C, D or E) to the estimated hypothesis ($\color{blue}\rightarrow$), at any given training cycle, and is represented by

$Cost Function = \sum_{i=1}^N(y_i-h_\theta(x_i))^2$

Here $(x_i, y_i)$ represents the data points, and $h_\theta(x_i)$ represents the best fit.

The minimum distance between a point (A, B, C, D or E) is represented by a perpendicular line drawn from that point on to the current best guess (green arrows).

The goal of least square function is to the define an objective function which when minimized would give rise to the least distance between the hypothesis and all the points combined, but won’t necessarily minimize the distance between the hypothesis and a single input point.

**Question**

Why don’t we define the Cost Function for linear regression as the least distance between the input data point and the hypothesis (defined by a line perpendicular to the hypothesis) passing through the input datapoin, as given by ($\color{green}\rightarrow$)?

Answer 1

When you have noise in both the dependent variable (vertical errors) and the independent variable (horizontal errors), the least squares objective function can be modified to incorporate these horizontal errors. The problem in how to weight these two types of errors. This weighting usually depends on the ratio of the variances of the two errors:

1. If the variance of the vertical error is extremely large relative to the variance of the horizontal error, OLS is correct.
2. If the variance of the horizontal error is extremely large relative to the variance of the vertical error, inverse least squares (in which $x$ is regressed on $y$ and the inverse of the coefficient estimate for $y$ is used as the estimate of $\beta$) is appropriate.
3. If the ratio of the variance of the vertical error to the variance of the horizontal error is equal to the ratio of the variances of the dependent and independent variables, we have the case of "diagonal" regression, in which a consistent esti­mate turns out to be the geometric mean of the OLS and inverse least squares estimators.
4. If the ratio of these error variances is one, then we have the case of "orthogonal" regression, in which the sum of squared errors measured along a line perpendicular to the estimating line is minimized. This is what you had in mind.

In practice, the great drawback of this procedure is that the ratio of the error variances is not usually known and cannot usually be estimated, so the path forward is not clear.

Answer 2

One reason is that $$\sum_{i=1}^N(y_i-h_\theta(x_i))^2$$ is relatively easy to compute and optimize, while the proposed cost $$\sum_{i=1}^N \min_{x,y}\big[(y_i-h_\theta(x))^2+(x_i-x)^2\big]$$ has a nested minimization problem which may be quite hard depending on the choice of family for $h_\theta(x)$.

Answer 3

At the risk of being prosaic, the reason for the error function is that the standard interpretation is that the x is given and one is trying to best describe (or predict) the y component. So there is no error in the 'x'. For example you might try and understand (or predict) the closing price of a stock tomorrow based on today's closing price . Similarly one could try and understand the average temperature tomorrow in terms of today's average temperature. Obviously these examples are simple minded, but that is the idea. Incidentally something most people don't realize, but I think is clear from your examples, is that if one regresses y against x the regression line doesn't have to have any particular resemblance to the regression of x against y. Orthogonal regression is the term for a regression where one tries to find the line that minimizes the distance of points from a line. For example if one was trying to understand the relationship between the price of IBM stock and the price of AAPL stock, that would be the appropriate method.

Answer 4

The oversimplified version is that X is assumed to have no error. So if you look at point E in your plot for example, it's assumed that its X coordinate is precisely accurate. Typically this is the case when we can control X, in other words when we can set it to a specific value. In that case, the only error that can exist is in the Y direction, and that's why the error / cost function only includes the Y direction.

Whenever that's not the case, whenever we don't control X and X can have error, people do incorporate the X direction in the error function in something called type II or model II regression, and its variants. It can be tricky to do this if X and Y have different scales, so then you have to think about normalizations and such.

Answer 5

You are right that, when fitting a line through points, the orthogonal distance is the most natural loss function that can be applied to arbitrary lines (note that the y-distance becomes meaningless for lines perpendicular to the x-axis). This problem is known under a number of names, e.g. "orthogonal regression", or (the most used term, AFAIK) "Principal Component Analysis" (PCA). For a discussion of this problem in arbitrary dimenstions, see

Späth: "Orthogonal least squares fitting with linear manifolds." Numerische Mathematik 48, pp. 441–445, 1986

As @aginensky already pointed out, the idea behind Linear Regression is not to fit a line through points, but to predict y-values for given x-values. That's why only the distance in y is used, which is the prediction accuracy.

Reformulating the problem of fitting a curve $\vec{x}(t)$ through points $\vec{p}_i$, $i=1\ldots N$ as a prediction problem makes things complicated, because the predictor $t$ is unknown and even to some degree arbitrary. For curves other than straight lines, this is still a problem that is subject to active research. One possible (incomplete) approach is described in the following article, which is incomplete because it does not provide a solution for finding an initial guess for the curve, but only how to iteratively improve such an initial guess:

Wang, Pottmann, Liu: "Fitting B-spline curves to point clouds by curvature-based squared distance minimization." ACM Transactions on Graphics 25.2, pp. 214-238, 2006