This question occurred to me when I sat in a public lecture on unsolved questions in mathematics. It is well known that there are still many unsolved mathematical questions out there. It made me thinking what the unsolved problems in statistics are. After spending some time on googleing this topic, I do not think there exist relatively detailed discussion on this question. Hence, I would really like to hear what people think about it. Where is statistics going as a discipline? Should we spend more time on improving theory or should we focus on how to analyse specific data collected from all kinds of scientific experiments? Any thought on this is greatly appreciated. Thank you!
In my opinion, having roamed in fringes of statistics near social sciences, statistics should talk more, and relate better, to other disciplines, and statisticians should spend more time on learning how to better communicate (a) what they are useful for, (b) what their findings mean in terms of that discipline, (c) why these other disciplines are better off working together with statisticians than without them. I don't know if the future of statistics depends on this, but there has been too many forgone opportunities in its short history, with other disciplines inventing their own statistical methods when statistics proper could not deliver. Nearly every other scientific/research discipline, from biology to anthropology, from psychiatry to structural engineering, can easily put a list of 5-10-20 open questions it would want statistics to answer.
David Cox explained it all in his interview.
@ocram pointed to Q14-15. Interestingly, I also found his answers enlightening. I was very sceptical of Big Data hype. Physicists dealt with enormous data sets for decades without much noise and annoying advertising, so did genetics researchers. Now once the marketing folks got involved it's Justin Bieber of statistics. However, Cox is right that in social sciences we never had large data sets available, with exception of quantitative finance, maybe. In fact, many econometrics techniques were specifically developed to deal with small samples. Thus it is interesting what will come out of Big Data push, maybe some exciting developments in statistics. I think emphasis would be on social sciences, where there are no good models of anything. Having bad models and little data could be quite different from having bad models and a lot of data, maybe there will be less focus on understanding of phenomena, in favor just getting the accurate forecasts through sheer volume of data and clever stats.
How to think about causal inference when there is treatment control interference or general equilibrium effects.