I think it's important to remember that different methods are good for different things, and significance testing isn't all there is in the world of statistics.
1 and 3) EB is probably not a valid hypothesis testing procedure, but it's also not meant to be.
Validity could be many things, but you're talking about Rigorous Experimental Design so we're probably discussing a hypothesis test that's supposed to help you make the right decision with a certain long-run frequency. This is a strictly dichotomous yes/no-type regime that's mostly useful for people who have to make a yes/no-type decision. There is a lot of classical work on this by very smart people indeed. These methods have nice theoretical validity in the limit assuming that all your assumptions hold, &c. However, EB certainly wasn't meant for this. If you want the machinery of the classical NHST methods, stick to the classical NHST methods.
2) EB is best applied in problems where you're estimating many similar, variable quantities.
Efron himself opens his book Large-Scale Inference listing three distinct eras of the history of statistics, pointing out that we are currently in
[the] era of scientific mass production, in which new technologies typified by the microarray allow a single team of scientists to produce data sets of a size Quetelet would envy. But now the flood of data is accompanied by a deluge of questions, perhaps thousands of estimates or hypothesis tests that the statistician is charged with answering together; not at all what the classical masters had in mind.
He goes on:
By their nature, empirical Bayes arguments combine frequentist and Bayesian elements in analyzing problems of repeated structure. Repeated structures are just what scientific mass production excels at, e.g., expression levels comparing sick and healthy subjects for thousands of genes at the same time by means of microarrays.
Perhaps the most successful recent application of EB is limma, available on Bioconductor. This is an R-package with methods for assessing differential expression (ie microarrays) between two study groups across tens of thousands of genes. Smyth shows their EB methods yield a t-statistic with more degrees of freedom than if you were to compute regular gene-wise t-statistics. The use of EB here "is equivalent to shrinkage of the estimated sample variances toward a pooled estimate, resulting in far more stable inference when the number of arrays is small," which is often the case.
As Efron points out above this isn't anything like what classical NHST was developed for, and the setting is usually more exploratory than confirmatory.
4) Generally you can see EB as a shrinkage method, and it can be useful everywhere that shrinkage is useful
The limma example above mentions shrinkage. Charles Stein gave us the astonishing result that when estimating the means for three or more things, there is an estimator that is better than using the observed means, $X_1, ..., X_k$. The James-Stein estimator has the form $\hat \theta^{JS}_i = (1- c/S^2)X_i,$ with $S^2=\sum_{j=1}^k X_j,$ and $c$ a constant. This estimator shrinks the observed means toward zero, and it is better than using $X_i$ in the strong sense of uniformly lower risk.
Efron and Morris showed a similar result for shrinking toward the pooled mean $\bar X,$ and this is what EB estimates tend to be. Below is an example I've made shrinking crime rates in different cities with EB methods. As you can see the more extreme estimates get shrunk a fair distance toward the mean. Smaller cities, where we can expect more variance, receive heavier shrinkage. The black point represents a large city, which has received basically no shrinkage. I have some simulations that show that these estimates do indeed have lower risk than using the observed MLE crime rates.
The more similar the quantites to be estimated, the more likely it is that shrinkage is useful. The book you refer to uses hit rates in baseball. Morris (1983) points to a handful of other applications:
- Revenue sharing---census bureau. Estimates per capita census income for several areas.
- Quality assurance---Bell Labs. Estimates number of failures for different time periods.
- Insurance rate-making. Estimates risk per exposure for groups of insured or for different territories.
- Law school admissions. Estimates weight for LSAT score relative to GPA for different schools.
- Fire alarms---NYC. Estimates false alarm rate for different alarm box locations.
These are all parallel-estimation problems and as far as I know they are more about making a good prediction of what a certain quantity is than they are about figuring out a yes/no decision.
Some references
- Efron, B. (2012). Large-scale inference: empirical Bayes methods for estimation, testing, and prediction (Vol. 1). Cambridge University Press.
Chicago
- Efron, B., & Morris, C. (1973). Stein's estimation rule and its competitors—an empirical Bayes approach. Journal of the American Statistical Association, 68(341), 117-130.
Chicago
- James, W., & Stein, C. (1961, June). Estimation with quadratic loss. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability (Vol. 1, No. 1961, pp. 361-379).
Chicago
- Morris, C. N. (1983). Parametric empirical Bayes inference: theory and applications. Journal of the American Statistical Association, 78(381), 47-55.
- Smyth, G. K. (2004). Linear models and empirical Bayes methods for assessing differential expression in microarray experiments. Statistical Applications in Genetics and Molecular Biology Volume 3, Issue 1, Article 3.
