56
$\begingroup$

I still remember the Annals of Statistics paper on Boosting by Friedman-Hastie-Tibshirani, and the comments on that same issues by other authors (including Freund and Schapire). At that time, clearly Boosting was viewed as a breakthrough in many respects: computationally feasible, an ensemble method, with excellent yet mysterious performance. Around the same time, SVM came of age, offering a framework underpinned by solid theory and with plenty of variants and applications.

That was in the marvelous 90s. In the past 15 years, it seems to me that a lot of Statistics has been a cleaning and detailing operation, but with few truly new views.

So I'll ask two questions:

  1. Have I missed some revolutionary/seminal paper?
  2. If not, are there new approaches that you think have the potential to change the viewpoint of statistical inference?

Rules:

  1. One answer per post;
  2. References or links welcome.

P.S.: I have a couple of candidates for promising breakthroughs. I will post them later.

$\endgroup$
  • 5
    $\begingroup$ See stats.stackexchange.com/q/1883/159 for a similar question (which was closed as subjective & argumentative). $\endgroup$ – Rob Hyndman Jan 21 '11 at 3:00
  • 1
    $\begingroup$ I was about to bring up the same thread. Smells like a duplicate. $\endgroup$ – Dirk Eddelbuettel Jan 21 '11 at 3:03
  • 1
    $\begingroup$ It's subjective, sure, but isn't it still okay for CW? $\endgroup$ – Christopher Aden Jan 21 '11 at 6:53
  • 1
    $\begingroup$ That was on a longer time scale. I don't think it's a duplicate. As for argumentative, it's up to the participants. I am not trying to award a trophy here, just to keep abreast of seminal papers I and others may have missed. Since there is no right answer, I am all for a CW. I find it interesting that so far all the answers are on bayesian innovations. $\endgroup$ – gappy Jan 21 '11 at 15:39
  • 2
    $\begingroup$ This seems like a post that can be grandfathered. I think this can stay open. $\endgroup$ – gung - Reinstate Monica Dec 30 '15 at 15:50

13 Answers 13

43
$\begingroup$

The answer is so simple that i have to write all this gibberish to make CV let me post it: R

$\endgroup$
14
$\begingroup$

I'm not sure if you would call it a "breakthrough" per se, But the Publishing of Probability Theory: The Logic of Science By Edwin Jaynes and Larry Bretthorst may be noteworthy. Some of the things they do here are:

1) show equivalence between some iterative "seasonal adjustment" schemes and Bayesian "nuisance parameter" integration.

2) resolved the so called "Marginalisation Paradox" - thought to be the "death of bayesianism" by some, and the "death of improper priors" by others.

3) the idea that probability describes a state of knowledge about a proposition being true or false, as opposed to describing a physical property of the world.

The first three chapters of this book are available for free here.

$\endgroup$
  • 2
    $\begingroup$ Unfortunately, Jaynes's resolution of the marginalization paradox was flawed. See Kevin Van Horn's Notes on Jaynes's Treatment of the Marginalization Paradox, available here. $\endgroup$ – Cyan Mar 30 '12 at 2:06
  • 1
    $\begingroup$ @cyan - Note that while his resolution was flawed in some areas his underlying principles solved it. The general rule of proper priors and their convergent limits means the mp cannot arise. The flaw is most likely due to the book being unfinished over most of part two. I like the resolution [here]( arxiv.org/abs/math/0310006) better than ksvh version. shorter and more general. $\endgroup$ – probabilityislogic Mar 30 '12 at 11:41
14
$\begingroup$

As an applied statistician and occasional minor software author, I'd say:

WinBUGS (released 1997)

It's based on BUGS, which was released more than 15 years ago (1989), but it's WinBUGS that made Bayesian analysis of realistically complex models available to a far wider user base. See e.g. Lunn, Spiegelhalter, Thomas & Best (2009) (and the discussion on it in Statistics in Medicine vol. 28 issue 25).

$\endgroup$
  • 2
    $\begingroup$ How does this change now that Stan is out? $\endgroup$ – Ari B. Friedman Nov 18 '12 at 21:59
13
$\begingroup$

LARS gets my vote. It combines linear regression with variable selection. Algorithms to compute it usually give you a collection of $k$ linear models, the $i$th one of which has nonzero coefficients for only $i$ regressors, so you can easily look at models of different complexity.

$\endgroup$
  • $\begingroup$ Have You ever used LARS? I'm asking because I have never heard about it earlier and it sounds really interesting. The orginal article is a bit long (93 pages) so I'd like to get some opinion before I get deep into it. $\endgroup$ – Tomek Tarczynski Jan 29 '11 at 11:26
  • $\begingroup$ @Tomek Tarczynski: I have used it a small amount. There is a package in Matlab (I am sure there is one or more in R), which I have used. It also provides a sparse PCA, which I was more interested in. I admit I only skimmed the paper. ;) $\endgroup$ – shabbychef Jan 29 '11 at 18:35
11
$\begingroup$

The introduction of the "intrinsic discrepancy" loss function and other "parameterisation free" loss functions into decision theory. It has many other "nice" properties, but I think the best one is as follows:

if the best estimate of $\theta$ using the intrinsic discrepancy loss function is $\theta^{e}$, then the best estimate of any one-to-one function of $\theta$, say $g(\theta)$ is simply $g(\theta^{e})$.

I think this is very cool! (e.g. best estimate of log-odds is log(p/(1-p)), best estimate of variance is square of standard deviation, etc. etc.)

The catch? the intrinsic discrepancy can be quite difficult to work out! (it involves min() funcion, a likelihood ratio, and integrals!)

The "counter-catch"? you can "re-arrange" the problem so that it is easier to calculate!

The "counter-counter-catch"? figuring out how to "re-arrange" the problem can be difficult!

Here are some references I know of which use this loss function. While I very much like the "intrinsic estimation" parts of these papers/slides, I have some reservations about the "reference prior" approach that is also described.

Bayesian Hypothesis Testing:A Reference Approach

Intrinsic Estimation

Comparing Normal Means: New Methods for an Old Problem

Integrated Objective Bayesian Estimation and Hypothesis Testing

$\endgroup$
11
$\begingroup$

Just falling within the 15 year window, I believe, are the algorithms for controlling False Discovery Rate. I like the 'q-value' approach.

$\endgroup$
9
$\begingroup$

Adding my own 5 cents, I believe the most significant breakthrough of the past 15 years has been Compressed Sensing. LARS, LASSO, and a host of other algorithms fall in this domain, in that Compressed Sensing explains why they work and extends them to other domains.

$\endgroup$
  • 1
    $\begingroup$ I've looked at Compressed Sensing and as a non-statistician I keep asking myself, "Isn't this just inverse random projection?". I know that "just" is an easy word to throw around, but it feels like people are leaving out what seem like obvious connections between random projection (circa 2000) and compressed sensing (circa 2004). $\endgroup$ – Wayne Aug 17 '11 at 18:10
9
$\begingroup$

Something that has very little to do with statistics themselves, but has been massively beneficial: The increasing firepower of computers, making larger datasets and more complex statistical analysis more accessible, especially in applied fields.

$\endgroup$
8
$\begingroup$

The Expectation-Propagation algorithm for Bayesian inference, especially in Gaussian Process classification, was arguably a significant breakthrough, as it provides an efficient analytic approximation method that works almost as well as computationally expensive sampling based approaches (unlike the usual Laplace approximation). See the work of Thomas Minka and others on the EP roadmap

$\endgroup$
  • $\begingroup$ EP does seem cool (although it still hurts my head). Does it still lack general convergence guarantees? $\endgroup$ – conjugateprior Jan 31 '11 at 13:27
7
$\begingroup$

I think that the 'Approximate Bayesian Inference for Latent Gaussian Models Using Integrated Nested Laplace Approximations' of H. Rue et. al (2009) is a potential candidate.

$\endgroup$
  • $\begingroup$ We may include with this 2011's further developpements in relation with Stochastic Partial Differential Equations Lindgren, Rue and Lindström $\endgroup$ – Yves Jul 7 at 11:41
2
$\begingroup$

In my opinion, everything allowing you to run new models on a large scale is a breakthrough. Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP) could be a candidate (though the idea is new and there have not been many implementations of the idea presented).

$\endgroup$
2
$\begingroup$

While a bit more general than statistics, I think there have been important advances in methods of reproducible research (RR). For example the development of R's knittr and Sweave packages and "R Markdown" notebooks, LyX and LaTeX improvements have contributed significantly to data sharing, collaboration, verification/validation, and even additional statistical advancement. Refereed papers in statistical, medical, and epidemiological journals rarely allowed one to reproduce results easily prior to the emergence of these reproducible research methods/technologies. Now, several journals are requiring reproducible research and many statisticians are using RR and posting code, their results and data sources on the web. This has also helped to foster data science disciplines and made statistical learning more accessible.

$\endgroup$
1
$\begingroup$

In my opinion, paper published in 2011 in Science magazine. Authors propose very interesting measure of association between pair of random variables that works well in many situations where similar measures fail (Pearson, Spearman, Kendall). Really nice paper. Here it is.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.