Empirial Bayes for means

Question

I have data for US ~3100 counties where the variable is a mean score based on a sample. However, for many small counties, the sample size is quite small (like 5), and so these mean values fluctuate a lot and result in extreme values. I need to use this variable as an outcome in a regression and for plotting, so the fluctuations are a problem for me (I use weights in regression but for plotting purposes, the data has to be adjusted itself).

One seemingly common approach in spatial statistics is to use the empirical Bayes approach to move the values towards the global mean. However, the examples I can find of this approach all refer to count data, whereas my data are continuous.

• http://varianceexplained.org/r/empirical_bayes_baseball/
• https://mgimond.github.io/Spatial/mapping-rates-in-r.html#unstable-ratios

Example of spatially informed empirical Bayes adjustment from link 2.

Here's a simple fictive example in R (without the spatial component).

#simulate population sizes, true means, and observed means
library(tidyverse)
library(truncnorm)

#sim data
d = tibble(
  n = runif(1e4, min = 2, max = 100) %>% round(),
  true_mean = rnorm(1e4, mean = 100, sd = 15),
  observed_mean = NA
)

#sample the means
d$observed_mean = map2_dbl(d$n, d$true_mean, ~mean(rnorm(.x, mean = .y, sd = 15)))

#plot
ggplot(d, aes(true_mean, observed_mean, color = n)) + 
  geom_point(alpha = .5)

which gives this plot:

We see that the smaller samples tend to produce outlying values, which one would need to shrink somewhat towards the mean.

My questions are:

1. Is there a simple function for adjusting these values towards the mean in the same way as in the empirical Bayes examples?
2. Is there a spatially informative version of this approach as in the first link?

I am looking for an R solution.

Answer 1

Is there a simple function for adjusting these values towards the mean in the same way as in the empirical Bayes examples?

All proper prior distributions result in shrinkage estimators. All empirical Bayes methods take a value that you are not interested in from the data to create a prior on the estimator that you are interested in. For example, using the grand mean to estimate particular mean values.

There is some controversy regarding the use of this method because you are using the data twice, once to estimate the prior then again to estimate the parameter. It is preferable to only use data once.

A non-controversial way to accomplish the same thing in the batting average example would have been to either look at the prior season's grand average or even just grab a handful of batting averages from outside that season and average them. I am not an opponent of empirical Bayes, but I did think you should be aware of the potential issue.

A slight improvement over using Empirical Bayes is to look at any prior research article on the topic and use the parameter estimates to create priors. It isn't simple, but it adds accuracy though you will likely have to hand code your prior density.

If you decide to use Empirical Bayes, there are no built-in restrictions as to the type of data. It can be averages, ratios, counts, slopes or just about anything. The purpose of the prior is to restrict the search area of the parameter estimates.

There is one thing that is a bit deceptive about a baseball example. You are only estimating $\pi_i$, the probability for each batter. There is only one parameter per batter. Other distributions also have nuisance parameters such as the variance. You need a prior for the variance and covariance as well if you are doing regression.

You may want to flatten the marginal density of the variances and covariances from that indicated by the grand variance. A model made up of many components could have quite a bit of variability in the natural variances observed component-wise.

Is there a spatially informative version of this approach as in the first link?

Yes, empirical Bayes is not specific to any domain. With spatial data, however, you can use distance to influence your priors, particularly with covariances. An observation in Miami should influence Tampa more than Portland.

You should read the literature out there on choosing a prior density.

https://www2.stat.duke.edu/courses/Spring13/sta732.01/priors.pdf

and

How to choose prior in Bayesian parameter estimation

may help you start.

You do not need to directly weight the data for visualization. Instead, plot the expectation of the marginal posterior densities for each region.