I'd like to know whether I am misusing Pearson's chi-squared test. And if so, what should I be doing instead.
I've a game-playing program, a "bot", for a zero-sum two-player game. To improve it, many strategies are played against each other, iteratively finding better and better strategies. (Where "strategy" means an arbitrary function whose input is the game position, in particular various features extracted from it, and whose output is a number to be maximized.)
When testing strategy A vs B, many A versus B games are played, starting from random positions. Tests terminate when there is 95% confidence (using Pearson's chi-squared) that we can reject the null-hypothesis. Once a test terminates a small change is made to the winning strategy and a new test begins...
Binomial and this chi-squared test seem to be defined for rejecting the null-hypothesis, but that's not exactly what I am doing here. Let's say I am seeing more wins with B than A. Then rather than rejecting
WinFrequency(A) = WinFrequency(B), the null hypothesis, I am rejecting
WinFrequency(A) >= WinFrequency(B).
Given an infinite # of trials, we should always reach 95% confidence, one way or the other... an infinite # of times. Does this cast any additional doubt upon my results when there are "sufficiently large" number of tests and it is already believed that the difference between WinFrequency of A and B is no greater than some "sufficiently small" value? In short, I'm wanting a metric to determine when I should stop testing A and B. Ideally this would be parameterized - something like "stop testing when we are 95% certain that B has no more than X% advantage in WinFrequency.
Note: by "WinFrequency" I mean the frequency with which a strategy would beat another, given an infinite number of trials - the expected value. Also note that draws are impossible - this game always ends in a win/loss.