When would least squares be a bad idea? If I have a regression model:
$$ 
Y = X\beta + \varepsilon
$$
where  $\mathbb{V}[\varepsilon] = Id \in \mathcal{R} ^{n \times n}$   and $\mathbb{E}[\varepsilon]=(0, \ldots , 0)$,
when would using $\beta_{\text{OLS}}$, the ordinary least squares estimator of $\beta$, be a poor choice for an estimator?
I am trying to figure out an example were least squares works poorly. So I am looking for a distribution of the errors that satisfices previous hypothesis but yields bad results. If the family of the distribution would be determined by mean and variance that would be great. If not, it's OK too.
I know that "bad results" is a little vague, but I think the idea is understandable.
Just to avoid confusions, I know least squares are not optimal, and that there are better estimators like ridge regression. But that's not what I am aiming at. I want an example were least squares would be unnatural.
I can imagine things like, the error vector $\epsilon$ lives in a non-convex region of $\mathbb{R}^n$, but I'm not sure about that.
Edit 1: As an idea to help an answer (which I can't figure how to take further). $\beta_{\text{OLS}}$ is BLUE. So it might help to think about when a linear unbiased estimator would not be a good idea.
Edit 2: As Brian pointed out, if $XX'$ is bad conditioned, then $\beta_{\text{OLS}}$ is a bad idea because variance is too big, and Ridge Regression should be used instead.
I'm more interested is in knowing what distribution should $\varepsilon$ in order to make least squares work bad.
$\beta_{\text{OLS}} \sim \beta+(X'X)^{-1}X'\varepsilon$ Is there a distribution with zero mean and identity variance matrix for $\varepsilon$ that makes this estimator not efficient?
 A: One example would be where you do not want to estimate the mean. This came up in work I used to do where we were estimating the number of sex partners people had, as part of modelling the spread of HIV/AIDS. There was more interest in the tails of the distribution: Which people have many many partners?
In this case, you could want quantile regression; an underused method, in my opinion. 
A: If $X$ is a badly conditioned matrix or exactly singular, then your least squares estimator will be extremely unstable and useless in practice.  
If you limit your attention to the distribution of $\epsilon$, then you should keep in mind that the Gauss-Markov theorem ensures that the least squares solution will be a minimum variance unbiased estimator.  
However, if the distribution of $\epsilon$ is sufficiently extreme, then it's possible to construct examples where the distribution of the estimates has bad properties (in particular, the possibility (albeit with low probability) of extremely large errors in $\beta$) despite being minimum variance.  
