Designing Computer Science experiments (Hypotheses and errors?) For my master thesis I want to set-up several experiments. One thing my professor has been complaining about is that most computer science experiments are lacking scientifically. To not fall into this trap I want to properly design my experiments. Unfortunately I could not find much literature on CS experiment design. (Except for D. Feitelson's "Experimental Computer Science: The Need for a Cultural Change" available here which is more of an appeal than a guide.)
I remember that in statistics people often use a H_0 and H_1 hypothesis together with the Type I and Type II error. I wonder if this is applicable in the field of computer science.
A quick example that I could think of:

H_0: Both methods create the same output in this scenario 
H_1: One method creates a worse output in this scenario (worse: according to
  metric X)

But then how would I calculate the Type I and Type II errors? In a speed-benchmark I could run the experiments multiple times and then possibly calculate how likely it is that the benchmark result is still wrong. However often I just want to compare the output of two deterministic methods. Measuring the quality according to a certain metric. This result doesn't change when running the experiment multiple times. What do I do then? Is the null hypothesis approach not a good fit for these kind of experiments?
 A: You should take a look at Catherine McGeoch's book, "A Guide To Experimental Algorithmics"  McGeoch has been criticized for her work on a study of the performance of the D-wave quantum computer, but I think the book is a good source.  
http://www.amazon.com/Guide-Experimental-Algorithmics-Catherine-McGeoch/dp/0521173019
A: I don't know exactly what kind of computer experiments you are trying to do. If you are trying to judge the performance of algorithms, there is now a field called experimental algorithmics which has a textbook, journals, and conferences. If you're comparing the performance of two or more algorithms on different inputs, you should probably look at references from this field; this is exactly the kind of thing they study, and they have figured out ways to do this which work much better than the naive first approach that you might think of. Furthermore, the fact that you are following the techniques of an established subfield of computer science should address your professor's complaints that the experiments are lacking scientifically. Even if the experimental design you come up with after looking at the techniques in this field is the same as the one you would have come up on your own, your professor is less likely to object to it. 
If you just have two algorithms you want to compare, one input, and one output for each algorithm, there is very little statistical analysis you can do, as this scenario gives you only two data points, and without some kind of model for the behavior of these algorithms, there is absolutely nothing you can do statistically with two data points.  
