Why squared residuals instead of absolute residuals in OLS estimation? Why are we using the squared residuals instead of the absolute residuals in OLS estimation?
My idea was that we use the square of the error values, so that residuals below the fitted line (which are then negative), would still have to be able to be added up to the positive errors. Otherwise, we could have an error of 0 simply because a huge positive error could cancel with a huge negative error.
So why do we square it, instead of just taking the absolute value? Is that because of the extra penalty for higher errors (instead of 2 being 2 times the error of 1, it is 4 times the error of 1 when we square it).
 A: I can't help quoting from Huber, Robust Statistics, p.10 on this (sorry the quote is too long to fit in a comment):

Two time-honored measures of scatter are the mean absolute deviation
$$d_n=\frac{1}{n}\sum|x_i-\bar{x}|$$
and the mean square deviation
$$s_n=\left[\frac{1}{n}\sum(x_i-\bar{x})^2\right]^{1/2}$$
There was a dispute between Eddington (1914, p.147) and Fisher  (1920,
  footnote on p. 762) about the relative merits of $d_n$  and
  $s_n$.[...] Fisher seemingly settled the matter by pointing  out that
  for normal observations $s_n$ is about 12% more efficient  than $d_n$.

By the relation between the conditional mean $\hat{y}$ and the unconditional 
mean $\bar{x}$ a similar argument applies to the residuals.
A: Both are done.
Least squares is easier, and the fact that for independent random variables "variances add" means that it's considerably more convenient; for examples, the ability to partition variances is particularly handy for comparing nested models. It's somewhat more efficient at the normal (least squares is maximum likelihood), which might seem to be a good justification -- however, some robust estimators with high breakdown can have surprisingly high efficiency at the normal.
But L1 norms are certainly used for regression problems and these days relatively often.
If you use R, you might find the discussion in section 5 here useful:
https://socialsciences.mcmaster.ca/jfox/Books/Companion/appendices/Appendix-Robust-Regression.pdf
(though the stuff before it on M estimation is also relevant, since it's also a special case of that)
A: One thing that has not been mentioned yet is uniqueness.  The least squares approach always produces a single "best" answer if the matrix of explanatory variables is full rank.  When minimizing the sum of the absolute value of the residuals it is possible that there may be an infinite number of lines that all have the same sum of absolute residuals (the minimum).  Which of those line should be used?
A: When the problem is expressed stochastically: $Y=aX+b+\epsilon$, where $\epsilon$ is normally distributed, the maximum likelihood estimate is the OLS estimate - not the minimum absolute deviation (MAD) estimate. So that's nice.
Furthermore, there is a strong link between OLS estimation and linear algebra. $\hat{Y}$ is a linear function of $Y$ --- in fact, it is a projection onto a subspace defined by the independent variables.
A lot of nice things happen with OLS --- MAD, not so much. And as @user603 points out, OLS are more efficient (where the normal model holds). It is less robust, of course.
