Why does OLS estimation involve taking vertical deviations of the points to the line rather than horizontal distances?
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ Old question, but I think viewing the data as a sample of a probability distribution that is parametrised by x is useful $\endgroup$– BendyFeb 19, 2019 at 10:51
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
2
OLS (ordinary least squares) assumes that the values represented by the horizontal distances are either predetermined by the experimenter or measured with high accuracy (relative to the vertical distances). When there is a question of uncertainty in the horizontal distances, you shouldn't be using OLS, but instead should look into errors-in-variables models or, possibly, principal components analysis.
-
$\begingroup$ "Orthogonal regression" is another thing one might encounter when looking for methods to deal with contaminated abscissas and ordinates. $\endgroup$ Oct 23, 2010 at 2:31
-
$\begingroup$ +1 This is yet a niche only in statistics; more complex least squares methods (not only adding X variability but also different penalties for points based on error approximations) are common in experimental physics; ROOT framework has dozens of such. $\endgroup$– user88Oct 23, 2010 at 11:38
$\begingroup$
$\endgroup$
Interesting Question. My answer would be that when we are fitting an OLS model we are implicitly and primarily trying to predict/explain the dependent variable at hand - the "Y" in the "Y vs X." As such, our main concern would be to minimize the distance from our fitted line to the actual observations with respect to the outcome, which means minimizing the vertical distance. This of course defines the residuals.
Also, least squares formulas are easier to derive than most other competing methods, which is perhaps why it came around first. :P
As 'whuber' alludes to above, there are other approaches that treat X and Y with equal emphasis when fitting a best-fit line. One such approach that I'm aware of is "principal lines" or "principal curves" regression, which minimizes the orthogonal distances between the points and the line (instead of a vertical error lines you have ones at 90 degrees to the fitted line). I post one reference below for your reading. It's lengthy but very accessible and enlightening.
Hope this helps, Brenden
- Trevor Hastie. Principal Curves and Surfaces, PhD thesis, Stanford University; 1984
$\begingroup$
$\endgroup$
It possibly also relates to designed experiments - if x is a controlled quantity that is part of the experimental design, it is treated as deterministic; whilst y is the outcome, and is a random quantity. x might be a continuous quantity (eg concentration of some drug) but could be a 0/1 split (leading to a 2 sample t-test assuming y is Gaussian). If x is a continuous quantity there may be some measurement error, but typically if this is much smaller than the variability of y then this is ignored.
