Why does OLS estimation involve taking vertical deviations of the points to the line rather than horizontal distances?
OLS (ordinary least squares) assumes that the values represented by the horizontal distances are either predetermined by the experimenter or measured with high accuracy (relative to the vertical distances). When there is a question of uncertainty in the horizontal distances, you shouldn't be using OLS, but instead should look into errors-in-variables models or, possibly, principal components analysis.
Interesting Question. My answer would be that when we are fitting an OLS model we are implicitly and primarily trying to predict/explain the dependent variable at hand - the "Y" in the "Y vs X." As such, our main concern would be to minimize the distance from our fitted line to the actual observations with respect to the outcome, which means minimizing the vertical distance. This of course defines the residuals.
Also, least squares formulas are easier to derive than most other competing methods, which is perhaps why it came around first. :P
As 'whuber' alludes to above, there are other approaches that treat X and Y with equal emphasis when fitting a best-fit line. One such approach that I'm aware of is "principal lines" or "principal curves" regression, which minimizes the orthogonal distances between the points and the line (instead of a vertical error lines you have ones at 90 degrees to the fitted line). I post one reference below for your reading. It's lengthy but very accessible and enlightening.
Hope this helps, Brenden
- Trevor Hastie. Principal Curves and Surfaces, PhD thesis, Stanford University; 1984
It possibly also relates to designed experiments - if x is a controlled quantity that is part of the experimental design, it is treated as deterministic; whilst y is the outcome, and is a random quantity. x might be a continuous quantity (eg concentration of some drug) but could be a 0/1 split (leading to a 2 sample t-test assuming y is Gaussian). If x is a continuous quantity there may be some measurement error, but typically if this is much smaller than the variability of y then this is ignored.