Okay so I have a function in the shape of a black box that I'd like to test. It's characteristics are:
- It gives either 0, 1 or -1 as output.
- I can query it easily, producing tens of thousands of samples in short time.
- I have no other prior information. I don't know the mean nor the variance.
My hypothesis is this: I want to test that it is "unbiased", in the sense that its mean is 0.
But I want to bias my testing in this way:
- False positives (i.e. Type II errors) must be minimized.
For this p-value testing (at least as I understand it) won't work, since it tries to minimize false negatives. My project needs to reject my implementation of this function every time that it truly has a bias: if it's biased I should reject it, but if it isn't then it's okay to reject it with a low-ish probability.
What should I be doing? The textbook I'm using (Ang & Tang for Engineering) explicitly skips over Type II error mitigation, so I have no idea on how to proceed.
Edit: more details on the problem, since it seems I can't quite explain it myself.
My function receives a point in N dimensions, and generates the gradients of a Noise function that belongs to the family of Perlin Noise. These gradients are unitary, and are built in such a way that the values of every coordinate are either α or -α (at "random"), and then one of them is chosen at "random" as well and made 0. α depends on the number of dimensions, but suffice to say that it's built to make the gradient unitary.
By "random" I mean a deterministic process that for any given point, always produces the same results. This is an important property of the Noise functions. But it also must seem random, and that's where things get complicated. The process is as follows:
- Point gets feed into a hash function.
- Hash of the point is used as the seed for a pseudorandom generator.
- Generator is used to produce all "random" actions.
I can, and am, using now a cryptographic hash function and a cryptographic pseudorandom generator. Those should have no bias I could measure (at least in polynomial time), and therefore I'm more or less certain that my function is indeed unbiased if I give it random input (for which I'm also using a cryptographic pseudorandom generator). But for sanity, and in case I then decide to try faster hashers or generators, I want a test that can check if the unbiasedness holds (with a certain probability of failure).
To finalize: I'm more interested in rejecting biased functions than in accepting unbiased ones. That's what I don't know how to do.