I'm a computer scientist working in data mining. It's no secret to say that computer scientists are fairly poor at doing systematic experimental design and evaluation - the use of p-values and confidence estimates is considered advanced :).

What I'd like to know if there are good courses/material to teach computer scientists about good experimental design. To make this more specific, I'll add the following information:

  • The course should be targeted at graduate students who can be assumed to have a reasonable understanding of probability, but limited background in statistics.
  • The course should focus on experimental design in "uncontrolled unnatural settings": in other words there is neither an underlying physical ground truth or a way to control the data gathering process (as with human subjects). Of course a good course will focus on fundamentals, but it should deal with this scenario in a significant way.
  • A computational element would be a bonus but is not mandatory. We deal with lots of data, but can figure out computational issues ourselves if need be.
  • 1
    $\begingroup$ All the conditions of the experiment you describe remind me of A/B-Tests ... coincidence ? :) $\endgroup$ – steffen May 29 '12 at 8:29

[Noah Smith][1] and [David Smith][2] offered a course sometime ago at JHU with similar motivations.


  • Lecture 1: introduction, review of statistics, hypothesis testing, sampling
  • Lecture 2: statistics of interest: means, quantiles, variance
  • Lectures 3–4: experiments with runtime and “space”
  • Lecture 5: exploratory data analysis
  • Lecture 6: parametric modeling, regression, and classification
  • Lecture 7: statistical debugging and profiling
  • Lecture 8: summary and review

For details, see Empirical Research Methods in Computer Science (600.408) http://www.cs.jhu.edu/~nasmith/erm/


I could sugest you two books instead of courses

The first one, as an application to bioinformatics and the second one for any discipline


Good question. I am keen to see the responses.

From a statistical standpoint two issues need addressing: most statistics and statistical designs discuss small sample statistics and most methodologies used by engineers are not "modern" statistics.

I have no immediate suggestion for the first problem beyond good schooling in data mining/exploration and the meaning of statistically different when faced with analysis of population (or large sample) statistics.

However two books of interest for introducing students to statistics would be from Rand Wilcox (a psychologist):

Wilcox, R. R. (2012). Introduction to Robust Estimation and Hypothesis Testing, 3rd Ed. Academic Press.

Wilcox, R. R. (2010). Fundamentals of Modern Statistical Methods: Substantially Improving Power and Accuracy, Springer, 2nd Ed.

  • 2
    $\begingroup$ It does seem to me that the first issue is one for research, and might not have "best practices" yet. It may very well be that a solid introduction to basic testing and drilling in the multiple hypothesis problem might be the best place to start. $\endgroup$ – Suresh Venkatasubramanian May 29 '12 at 15:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.