What are the 'big problems' in statistics?

Question

Mathematics has its famous Millennium Problems (and, historically, Hilbert's 23), questions that helped to shape the direction of the field.

I have little idea, though, what the Riemann Hypotheses and P vs. NP's of statistics would be.

So, what are the overarching open questions in statistics?

Edited to add: As an example of the general spirit (if not quite specificity) of answer I'm looking for, I found a "Hilbert's 23"-inspired lecture by David Donoho at a "Math Challenges of the 21st Century" conference: High-Dimensional Data Analysis: The Curses and Blessings of Dimensionality

So a potential answer could talk about big data and why it's important, the types of statistical challenges high-dimensional data poses, and methods that need to be developed or questions that need to be answered in order to help solve the problem.

Answer 1

A big question should involve key issues of statistical methodology or, because statistics is entirely about applications, it should concern how statistics is used with problems important to society.

This characterization suggests the following should be included in any consideration of big problems:

• How best to conduct drug trials. Currently, classical hypothesis testing requires many formal phases of study. In later (confirmatory) phases, the economic and ethical issues loom large. Can we do better? Do we have to put hundreds or thousands of sick people into control groups and keep them there until the end of a study, for example, or can we find better ways to identify treatments that really work and deliver them to members of the trial (and others) sooner?

• Coping with scientific publication bias. Negative results are published much less simply because they just don't attain a magic p-value. All branches of science need to find better ways to bring scientifically important, not just statistically significant, results to light. (The multiple comparisons problem and coping with high-dimensional data are subcategories of this problem.)

• Probing the limits of statistical methods and their interfaces with machine learning and machine cognition. Inevitable advances in computing technology will make true AI accessible in our lifetimes. How are we going to program artificial brains? What role might statistical thinking and statistical learning have in creating these advances? How can statisticians help in thinking about artificial cognition, artificial learning, in exploring their limitations, and making advances?

• Developing better ways to analyze geospatial data. It is often claimed that the majority, or vast majority, of databases contain locational references. Soon many people and devices will be located in real time with GPS and cell phone technologies. Statistical methods to analyze and exploit spatial data are really just in their infancy (and seem to be relegated to GIS and spatial software which is typically used by non-statisticians).

Answer 2

Michael Jordan has a short article called What are the Open Problems in Bayesian Statistics?, in which he polled a bunch of statisticians for their views on the open problems in statistics. I'll summarize (aka, copy-and-paste) a bit here, but it's probably best just to read the original.

Nonparametrics and semiparametrics

• For what problems is Bayesian nonparametrics useful and worth the trouble?
• David Dunson: "Nonparametric Bayes models involve infinitely many parameters and priors are typically chosen for convenience with hyperparameters set at seemingly reasonable values with no proper objective or subjective justification."
• "It was noted by several people that one of the appealing applications of frequentist nonparametrics is to semiparametric inference, where the nonparametric component of the model is a nuisance parameter. These people felt that it would be desirable to flesh out the (frequentist) theory of Bayesian semiparametrics."

Priors

• "Elicitation remains a major source of open problems."
• 'Aad van der Vaart turned objective Bayes on its head and pointed to a lack of theory for "situations where one wants the prior to come through in the posterior" as opposed to "merely providing a Bayesian approach to smoothing."'

Bayesian/frequentist relationships

• "Many respondents expressed a desire to further hammer out Bayesian/frequentist relationships. This was most commonly evinced in the context of high-dimensional models and data, where not only are subjective approaches to specification of priors difficult to implement but priors of convenience can be (highly) misleading."
• 'Some respondents pined for non-asymptotic theory that might reveal more fully the putative advantages of Bayesian methods; e.g., David Dunson: "Often, the frequentist optimal rate is obtained by procedures that clearly do much worse in finite samples than Bayesian approaches."'

Computation and statistics

• Alan Gelfand: "If MCMC is no longer viable for the problems people want to address, then what is the role of INLA, of variational methods, of ABC approaches?"
• "Several respondents asked for a more thorough integration of computational science and statistical science, noting that the set of inferences that one can reach in any given situation are jointly a function of the model, the prior, the data and the computational resources, and wishing for more explicit management of the tradeoffs among these quantities. Indeed, Rob Kass raised the possibility of a notion of “inferential solvability,” where some problems are understood to be beyond hope (e.g., model selection in regression where “for modest amounts of data subject to nontrivial noise it is im- possible to get useful confidence intervals about regression coefficients when there are large numbers of variables whose presence or absence in the model is unspecified a priori”) and where there are other problems (“certain functionals for which useful con- fidence intervals exist”) for which there is hope."
• "Several respondents, while apologizing for a certain vagueness, expressed a feeling that a large amount of data does not necessarily imply a large amount of computation; rather, that somehow the inferential strength present in large data should transfer to the algorithm and make it possible to make do with fewer computational steps to achieve a satisfactory (approximate) inferential solution."

Model Selection and Hypothesis Testing

• George Casella: "We now do model selection but Bayesians don’t seem to worry about the properties of basing inference on the selected model. What if it is wrong? What are the consequences of setting up credible regions for a certain parameter $β_1$ when you have selected the wrong model? Can we have procedures with some sort of guarantee?"
• Need for more work on decision-theoretic foundations in model selection.
• David Spiegelhalter: "How best to make checks for prior/data conflict an integral part of Bayesian analysis?"
• Andrew Gelman: "For model checking, a key open problem is developing graphical tools for understanding and comparing models. Graphics is not just for raw data; rather, complex Bayesian models give opportunity for better and more effective exploratory data analysis."

Answer 3

I'm not sure how big they are, but there is a Wikipedia page for unsolved problems in statistics. Their list includes:

Inference and testing

• Systematic errors
• Admissability of the Graybill–Deal estimator
• Combining dependent p-values in Meta-analysis
• Behrens–Fisher problem
• Multiple comparisons
• Open problems in Bayesian statistics

Experimental design

• Problems in Latin squares

Problems of a more philosophical nature

• Sampling of species problem
• Doomsday argument
• Exchange paradox

Answer 4

As an example of the general spirit (if not quite specificity) of answer I'm looking for, I found a "Hilbert's 23"-inspired lecture by David Donoho at a "Math Challenges of the 21st Century" conference:

High-Dimensional Data Analysis: The Curses and Blessings of Dimensionality

Answer 5

Mathoverflow has a similar question about big problems in probability theory.

It would appear from that page that the biggest questions are to do with self avoiding random walks and percolations.

Answer 6

You might check out Harvard's "Hard Problems in the Social Sciences' colloquium held earlier this year. Several of these talks offer issues in the use of statistics and modeling in the social sciences.

Answer 7

My answer would be the struggle between frequentist and Bayesian statistics. When people ask you which you "believe in", this is not good! Especially for a scientific discipline.