# What are the breakthroughs in Statistics of the past 15 years?

I still remember the Annals of Statistics paper on Boosting by Friedman-Hastie-Tibshirani, and the comments on that same issues by other authors (including Freund and Schapire). At that time, clearly Boosting was viewed as a breakthrough in many respects: computationally feasible, an ensemble method, with excellent yet mysterious performance. Around the same time, SVM came of age, offering a framework underpinned by solid theory and with plenty of variants and applications.

That was in the marvelous 90s. In the past 15 years, it seems to me that a lot of Statistics has been a cleaning and detailing operation, but with few truly new views.

So I'll ask two questions:

1. Have I missed some revolutionary/seminal paper?
2. If not, are there new approaches that you think have the potential to change the viewpoint of statistical inference?

Rules:

1. One answer per post;
2. References or links welcome.

P.S.: I have a couple of candidates for promising breakthroughs. I will post them later.

-
See stats.stackexchange.com/q/1883/159 for a similar question (which was closed as subjective & argumentative). –  Rob Hyndman Jan 21 '11 at 3:00
I was about to bring up the same thread. Smells like a duplicate. –  Dirk Eddelbuettel Jan 21 '11 at 3:03
It's subjective, sure, but isn't it still okay for CW? –  Christopher Aden Jan 21 '11 at 6:53
That was on a longer time scale. I don't think it's a duplicate. As for argumentative, it's up to the participants. I am not trying to award a trophy here, just to keep abreast of seminal papers I and others may have missed. Since there is no right answer, I am all for a CW. I find it interesting that so far all the answers are on bayesian innovations. –  gappy Jan 21 '11 at 15:39
I liked the 50-year question. I don't think it needed to be close. I am just not so sure we needed a new question. <Shrug> –  Dirk Eddelbuettel Jan 21 '11 at 18:15

The answer is so simple that i have to write all this gibberish to make CV let me post it: R

-

As an applied statistician and occasional minor software author, i'd say:

WinBUGS (released 1997)

It's based on BUGS, which was released more than 15 years ago (1989), but it's WinBUGS that made Bayesian analysis of realistically complex models available to a far wider user base. See e.g. Lunn, Spiegelhalter, Thomas & Best (2009) (and the discussion on it in Statistics in Medicine vol. 28 issue 25).

-
How does this change now that Stan is out? –  Ari B. Friedman Nov 18 '12 at 21:59

I'm not sure if you would call it a "breakthrough" per se, But the Publishing of Probability Theory: The Logic of Science By Edwin Jaynes and Larry Bretthorst may be noteworthy. Some of the things they do here are:

1) show equivalence between some iterative "seasonal adjustment" schemes and Bayesian "nuisance parameter" integration.

2) resolved the so called "Marginalisation Paradox" - thought to be the "death of bayesianism" by some, and the "death of improper priors" by others.

3) the idea that probability describes a state of knowledge about a proposition being true or false, as opposed to describing a physical property of the world.

The first three chapters of this book are available for free here.

-
Unfortunately, Jaynes's resolution of the marginalization paradox was flawed. See Kevin Van Horn's Notes on Jaynes's Treatment of the Marginalization Paradox, available here. –  Cyan Mar 30 '12 at 2:06
@cyan - Note that while his resolution was flawed in some areas his underlying principles solved it. The general rule of proper priors and their convergent limits means the mp cannot arise. The flaw is most likely due to the book being unfinished over most of part two. I like the resolution [here]( arxiv.org/abs/math/0310006) better than ksvh version. shorter and more general. –  probabilityislogic Mar 30 '12 at 11:41

LARS gets my vote. It combines linear regression with variable selection. Algorithms to compute it usually give you a collection of $k$ linear models, the $i$th one of which has nonzero coefficients for only $i$ regressors, so you can easily look at models of different complexity.

-
Have You ever used LARS? I'm asking because I have never heard about it earlier and it sounds really interesting. The orginal article is a bit long (93 pages) so I'd like to get some opinion before I get deep into it. –  Tomek Tarczynski Jan 29 '11 at 11:26
@Tomek Tarczynski: I have used it a small amount. There is a package in Matlab (I am sure there is one or more in R), which I have used. It also provides a sparse PCA, which I was more interested in. I admit I only skimmed the paper. ;) –  shabbychef Jan 29 '11 at 18:35

Something that has very little to do with statistics themselves, but has been massively beneficial: The increasing firepower of computers, making larger datasets and more complex statistical analysis more accessable, especially in applied fields.

-

Just falling within the 15 year window, I believe, are the algorithms for controlling False Discovery Rate. I like the 'q-value' approach.

-
Hmm, the much-cited Benjamini-Hochberg JRSSB paper was published in 1995, so just outside the window I'm afraid! jstor.org/stable/2346101 Storey's paper that introduced $q$-values was 2002 though. dx.doi.org/10.1111/1467-9868.00346 –  onestop Jan 29 '11 at 20:11

Adding my own 5 cents, I believe the most significant breakthrough of the past 15 years has been Compressed Sensing. LARS, LASSO, and a host of other algorithms fall in this domain, in that Compressed Sensing explains why they work and extends them to other domains.

-
I've looked at Compressed Sensing and as a non-statistician I keep asking myself, "Isn't this just inverse random projection?". I know that "just" is an easy word to throw around, but it feels like people are leaving out what seem like obvious connections between random projection (circa 2000) and compressed sensing (circa 2004). –  Wayne Aug 17 '11 at 18:10

The Expectation-Propagation algorithm for Bayesian inference, especially in Gaussian Process classification, was arguably a significant breakthrough, as it provides an efficient analytic approximation method that works almost as well as computationally expensive sampling based approaches (unlike the usual Laplace approximation). See the work of Thomas Minka and others on the EP roadmap

-
EP does seem cool (although it still hurts my head). Does it still lack general convergence guarantees? –  conjugateprior Jan 31 '11 at 13:27

The introduction of the "intrinsic discrepancy" loss function and other "parameterisation free" loss functions into decision theory. It has many other "nice" properties, but I think the best one is as follows:

if the best estimate of $\theta$ using the intrinsic discrepancy loss function is $\theta^{e}$, then the best estimate of any one-to-one function of $\theta$, say $g(\theta)$ is simply $g(\theta^{e})$.

I think this is very cool! (e.g. best estimate of log-odds is log(p/(1-p)), best estimate of variance is square of standard deviation, etc. etc.)

The catch? the intrinsic discrepancy can be quite difficult to work out! (it involves min() funcion, a likelihood ratio, and integrals!)

The "counter-catch"? you can "re-arrange" the problem so that it is easier to calculate!

The "counter-counter-catch"? figuring out how to "re-arrange" the problem can be difficult!

Here are some references I know of which use this loss function. While I very much like the "intrinsic estimation" parts of these papers/slides, I have some reservations about the "reference prior" approach that is also described.

Bayesian Hypothesis Testing:A Reference Approach

Intrinsic Estimation

Comparing Normal Means: New Methods for an Old Problem

Integrated Objective Bayesian Estimation and Hypothesis Testing

-