What are the breakthroughs in Statistics of the past 15 years?

Question

I still remember the Annals of Statistics paper on Boosting by Friedman-Hastie-Tibshirani, and the comments on that same issues by other authors (including Freund and Schapire). At that time, clearly Boosting was viewed as a breakthrough in many respects: computationally feasible, an ensemble method, with excellent yet mysterious performance. Around the same time, SVM came of age, offering a framework underpinned by solid theory and with plenty of variants and applications.

That was in the marvelous 90s. In the past 15 years, it seems to me that a lot of Statistics has been a cleaning and detailing operation, but with few truly new views.

So I'll ask two questions:

1. Have I missed some revolutionary/seminal paper?
2. If not, are there new approaches that you think have the potential to change the viewpoint of statistical inference?

Rules:

1. One answer per post;
2. References or links welcome.

P.S.: I have a couple of candidates for promising breakthroughs. I will post them later.

Answer 1

The answer is so simple that i have to write all this gibberish to make CV let me post it: R

Answer 2

I'm not sure if you would call it a "breakthrough" per se, But the Publishing of Probability Theory: The Logic of Science By Edwin Jaynes and Larry Bretthorst may be noteworthy. Some of the things they do here are:

1) show equivalence between some iterative "seasonal adjustment" schemes and Bayesian "nuisance parameter" integration.

2) resolved the so called "Marginalisation Paradox" - thought to be the "death of bayesianism" by some, and the "death of improper priors" by others.

3) the idea that probability describes a state of knowledge about a proposition being true or false, as opposed to describing a physical property of the world.

The first three chapters of this book are available for free here.

Answer 3

As an applied statistician and occasional minor software author, I'd say:

WinBUGS (released 1997)

It's based on BUGS, which was released more than 15 years ago (1989), but it's WinBUGS that made Bayesian analysis of realistically complex models available to a far wider user base. See e.g. Lunn, Spiegelhalter, Thomas & Best (2009) (and the discussion on it in Statistics in Medicine vol. 28 issue 25).

Answer 4

LARS gets my vote. It combines linear regression with variable selection. Algorithms to compute it usually give you a collection of $k$ linear models, the $i$th one of which has nonzero coefficients for only $i$ regressors, so you can easily look at models of different complexity.

Answer 5

The introduction of the "intrinsic discrepancy" loss function and other "parameterisation free" loss functions into decision theory. It has many other "nice" properties, but I think the best one is as follows:

if the best estimate of $\theta$ using the intrinsic discrepancy loss function is $\theta^{e}$, then the best estimate of any one-to-one function of $\theta$, say $g(\theta)$ is simply $g(\theta^{e})$.

I think this is very cool! (e.g. best estimate of log-odds is log(p/(1-p)), best estimate of variance is square of standard deviation, etc. etc.)

The catch? the intrinsic discrepancy can be quite difficult to work out! (it involves min() funcion, a likelihood ratio, and integrals!)

The "counter-catch"? you can "re-arrange" the problem so that it is easier to calculate!

The "counter-counter-catch"? figuring out how to "re-arrange" the problem can be difficult!

Here are some references I know of which use this loss function. While I very much like the "intrinsic estimation" parts of these papers/slides, I have some reservations about the "reference prior" approach that is also described.

Bayesian Hypothesis Testing:A Reference Approach

Intrinsic Estimation

Comparing Normal Means: New Methods for an Old Problem

Integrated Objective Bayesian Estimation and Hypothesis Testing

Answer 6

Just falling within the 15 year window, I believe, are the algorithms for controlling False Discovery Rate. I like the 'q-value' approach.

Answer 7

Adding my own 5 cents, I believe the most significant breakthrough of the past 15 years has been Compressed Sensing. LARS, LASSO, and a host of other algorithms fall in this domain, in that Compressed Sensing explains why they work and extends them to other domains.

Answer 8

Something that has very little to do with statistics themselves, but has been massively beneficial: The increasing firepower of computers, making larger datasets and more complex statistical analysis more accessible, especially in applied fields.

Answer 9

The Expectation-Propagation algorithm for Bayesian inference, especially in Gaussian Process classification, was arguably a significant breakthrough, as it provides an efficient analytic approximation method that works almost as well as computationally expensive sampling based approaches (unlike the usual Laplace approximation). See the work of Thomas Minka and others on the EP roadmap

Answer 10

I think that the 'Approximate Bayesian Inference for Latent Gaussian Models Using Integrated Nested Laplace Approximations' of H. Rue et. al (2009) is a potential candidate.

Answer 11

In my opinion, everything allowing you to run new models on a large scale is a breakthrough. Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP) could be a candidate (though the idea is new and there have not been many implementations of the idea presented).

Answer 12

While a bit more general than statistics, I think there have been important advances in methods of reproducible research (RR). For example the development of R's knittr and Sweave packages and "R Markdown" notebooks, LyX and LaTeX improvements have contributed significantly to data sharing, collaboration, verification/validation, and even additional statistical advancement. Refereed papers in statistical, medical, and epidemiological journals rarely allowed one to reproduce results easily prior to the emergence of these reproducible research methods/technologies. Now, several journals are requiring reproducible research and many statisticians are using RR and posting code, their results and data sources on the web. This has also helped to foster data science disciplines and made statistical learning more accessible.

Answer 13

In my opinion, paper published in 2011 in Science magazine. Authors propose very interesting measure of association between pair of random variables that works well in many situations where similar measures fail (Pearson, Spearman, Kendall). Really nice paper. Here it is.