A course in experimental design for data miners

Question

I'm a computer scientist working in data mining. It's no secret to say that computer scientists are fairly poor at doing systematic experimental design and evaluation - the use of p-values and confidence estimates is considered advanced :).

What I'd like to know if there are good courses/material to teach computer scientists about good experimental design. To make this more specific, I'll add the following information:

• The course should be targeted at graduate students who can be assumed to have a reasonable understanding of probability, but limited background in statistics.
• The course should focus on experimental design in "uncontrolled unnatural settings": in other words there is neither an underlying physical ground truth or a way to control the data gathering process (as with human subjects). Of course a good course will focus on fundamentals, but it should deal with this scenario in a significant way.
• A computational element would be a bonus but is not mandatory. We deal with lots of data, but can figure out computational issues ourselves if need be.

Answer 1

[Noah Smith][1] and [David Smith][2] offered a course sometime ago at JHU with similar motivations.

Outline:

• Lecture 1: introduction, review of statistics, hypothesis testing, sampling
• Lecture 2: statistics of interest: means, quantiles, variance
• Lectures 3–4: experiments with runtime and “space”
• Lecture 5: exploratory data analysis
• Lecture 6: parametric modeling, regression, and classification
• Lecture 7: statistical debugging and profiling
• Lecture 8: summary and review

For details, see Empirical Research Methods in Computer Science (600.408) http://www.cs.jhu.edu/~nasmith/erm/

Answer 2

I could sugest you two books instead of courses

• Computational Text Analysis For Functional Genomics and Bioinformatics by Soumya Raychaudhuri.
• The Oxford Handbook of Functional Data Analysis by Edited by Frederic Ferraty and Yves Romain.

The first one, as an application to bioinformatics and the second one for any discipline

Answer 3

Good question. I am keen to see the responses.

From a statistical standpoint two issues need addressing: most statistics and statistical designs discuss small sample statistics and most methodologies used by engineers are not "modern" statistics.

I have no immediate suggestion for the first problem beyond good schooling in data mining/exploration and the meaning of statistically different when faced with analysis of population (or large sample) statistics.

However two books of interest for introducing students to statistics would be from Rand Wilcox (a psychologist):

Wilcox, R. R. (2012). Introduction to Robust Estimation and Hypothesis Testing, 3rd Ed. Academic Press.

Wilcox, R. R. (2010). Fundamentals of Modern Statistical Methods: Substantially Improving Power and Accuracy, Springer, 2nd Ed.