Motivations for experiment design in statistical learning?

Question

My interests in statistics centre around statistical learning, including Bayesian inference, inference in combinatorial spaces, Monte Carlo methods, Markov decision processes, modeling stochastic processes and so on. It is mandatory at my university’s program that we take an experiment design course, which seems rather orthogonal to my interests.

I am hoping this opinion is a by product of my ignorance. Thus some questions I seek to answer include: what are applications of experiment design in statistical learning? How can I relate it to statistical learning? What are some motivating examples or challenging problems in experiment design?

Answer 1

This is an interesting question. The following stored google search gives many interesting hits, and both ways: Machine learning used in experimental design and experimental design used in machine learning.

Basically experimental design is about planning the collection of data. That must be useful in statistical learning/machine learning, as you can get much better results from your analysis with better data. One obvious application is planning of simulation experiments, as in this case the data collection is completely under your control.

You could do worse than start with this excellent book by Box, Hunter & Hunter. Look also at this list. This interesting-looking paper asks to rethink experimental design as algorithm design.

So use that required course to learn not only the classics, but also peek into some applications in the fields you mentioned, such as Bayesian experimental design, combinatorics?, Markov decision processes, stochastic processes.

Active learning seems to be a buzzword for combining learning with design ... reinforcement learning much the same! That viewpoint is supported by this Wikipedia article. Computerized adaptive testing can be seen as a forerunner of active learning, and is certainly using some experimental design. Some explanation of how that works can be found here: Statistical interpretation of Maximum Entropy Distribution.

While at it, the tag experiment-design covers many posts in here, too many still in need of answers&upvotes. So going trough that, answering, upvoting would be a great learning experience ...