Why we shouldn't be obsessed with unbiasedness In my Bayesian statistics class, my professor makes the remark that we should not be obsessed with unbiased estimator.
First: I understand this statement in the sense of trading biasedness for smaller variance, i.e. "good" estimator = estimator with smallest Mean Squared Error (MSE). This idea is seen among Frequentists also.
But second, and the confusing part: My professor then said something about an unbiased estimator is picking up noise and not signal. Since most of the observed data are just noise, we want to shrink our estimate to 0.
I do not quite understand what is noise and signal:
1) Don't all data points come from the same distribution, with the difference due only to sampling variation? 
2) Why can't unbiased estimator (say, sample mean of a normal model) differentiate noise and signal? If it can't, what sort of estimator can?
An example would be great, and a social science example would be best!
 A: Some people are annoyingly obsessed with unbiasedness. Bias and dispersion are measures of uncertainty which roughly correspond to accuracy and precision. You usually have a trade-off between accuracy and precision, some estimators may be more precise but less accurate and vice versa.
MSE is a sum of bias and the variance: $MSE=E[\beta-\hat{\beta}]^2=E[\beta-E[\hat{\beta}]+E[\hat{\beta}]-\hat{\beta}  ]^2=bias^2+Var[\hat{\beta}]$
Example. Let's say your parameter is from Breit Wiegner distribution.

There is no unbiased estimator. The bias is defined as $bias=E[\hat{\beta}]-\beta$, where $\beta$ - true value, and $\hat{\beta}$ is its estimator. In this case $E[\hat{\beta}]$ is not defined mathematically, so you can't compute the bias. 
This is an extreme example where unbiased estimator does not exist at all.
A: The motivation of the statement is a rejection of some very important work from Rao-Blackwell about the provision of so called UMVUEs (Uniform minimum variance unbiased estimators). This showed that there is a lower bound on variance of unbiased estimators, and if you achieve that, you usually get a very good estimator. Many of our most popular statistics are special cases of that, such as the z-test for normal data, or the least squares regression models (among linear estimators, which makes LS BLUE (best linear unbiased estimator)).
The big question is what happens when we stop caring about bias? Do we get better estimators? The answer is YES. For inference, plenty of biased estimators are extremely efficient at detecting associations in data. A great way of evaluating the quality of an estimator is with its MSE (mean squarred error), or:
$\mbox{MSE}(\hat{\theta}) = \mathcal{E}\left( (\hat{\theta} - \theta)^2\right)$
This can be rewritten as:
$\mbox{MSE}(\hat{\theta}) = \mathcal{E}\left( (\mathcal{E}(\hat{\theta}) - \theta)^2\right) + \mathcal{E} \left( (\mathcal{E}(\hat{\theta}) - \hat{\theta} )^2 \right)$
which is the sum of the squared bias and the variance of the estimator.
An example of an estimator that has better MSE than the UMVUE for a multivariate normal model is ridge regression. What your professor may have referred to regarding "shrink to zero" is shrinkate: the tendency for high dimensional statistics to have poor out-of-sample validity. The "squared error loss" (which is variance plus squared-bias) can be improved from using multivariate normal MLE. The ridge estimator uses an L2 penalty to penalize highly variable estimates. The LASSO uses an L1 penalty and also has shrinkage properties. The L1 penalty forces relatively small estimates to be exactly zero, but it is an abuse of terminology to claim they "shrink to zero" when shrinkage refers to optimizing MSE. Rather, we constrain.
http://www.few.vu.nl/~wvanwie/Courses/HighdimensionalDataAnalysis/WNvanWieringen_HDDA_Lecture234_RidgeRegression_20182019.pdf
Bayes estimators which are posterior means using a conjugate prior minimize squared error loss. 
Bearing that in mind, if MSE was all we called about controlling for, you can come up with a plethora of estimators which are a little biased but with much smaller variance than the UMVUE. And given that UMVUE estimators can have complicated asymptotic distributions, or are sometimes difficult to find, or sometimes don't exist, we often seek out biased estimators that lack all those problems.
