I have a script which randomly generates test cases for a certain data type and throws these at an algorithm which in turn returns a positive or negative result.

From this I simple calculate positive %.

So I get something like

Type1 ~ 80%; Type2 ~ 50%

The problem is I am not sure what is the standard deviation here or how confident I am with my results. Obviously n = number of test cases is important but knowing n to be say 1000, can I have a statistical test to see if if is reasonable to say that Type1 and Type2 are performing differently against the algorithm in question.

I am also curious if n = 1000 makes more sense than say n = 50 ran 20 times and taking the mean result.


1 Answer 1


With the absolute numbers you can construct a contingency table and use chi-square test of independence to compare 80% in Type A with 50% in Type B.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.