I work helping analyze large amounts of sequence data and I am trying to learn more design of experiments. However, something that I have observed is that, even though design of experiments is very important in some - smaller scale - realms, E.g. deciding the correlation structure of the data (RCBD, split-plot, etc) and the design of the experiment itself (fractional factorial, number of samples, etc), it's not formally taught much nor given much importance in big data (it's my impression), at least in the online courses. Is it just because it is not a "hot field" or because it is not as important? Any suggestions about how to go about learning more?? Especially as it applies to genomic data?
1
-
$\begingroup$ Have a look at the list in stats.stackexchange.com/questions/1815/… (and serach this site for design of experiments). Others: stats.stackexchange.com/questions/179067/… $\endgroup$– kjetil b halvorsen ♦Jan 16, 2016 at 21:39
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
Experimental design and Big data analytics are essential to various fields of applications. However, the two fields are usually considered at different phases of the work/research,etc...
Experimental Design answers the question of "how" to get your data through selection of a sampling strategy, suggestion of sampling locations and identification of significant factors to maximize the information gain about the process of interest. Therefore, Experimental design is mostly regarded as a pre-data generation process. Some researchers are also concerned with the so-called "Sequential design of experiments" which updates your design "sequentially" as more data is available.
On the other hand, Big data analytics and techniques are usually regarded as a post-data generation process. The critical role of big data analytics is to develop statistical-based models to infer useful information from the generated data such as estimation of parameters, point predictions, uncertainty quantification and others.
Regarding your question about how to learn more about design of experiments. I recommend two books:
- Design and Analysis of Experiments By Douglas Montgomery (which is an introductory level textbook)
- Experiments: Planning, Analysis and Optimization by Jeff Wu (which is a more advanced, graduate-level textbook)
-
1$\begingroup$ I would not recommend the first text above (Montgomery). A much better introduction is Box, Hunter & Hunter: Statistics for experimenters. $\endgroup$ Sep 8, 2016 at 12:58