The problem: I have N (~500) black boxes which each receive an input and output a noisy reward signal. The reward signal is non-stationary and heteroscedastic, but can be assumed stationary over short time samples. The population as a whole experience similar trends, but individuals are capable of wildly different patterns.
I have a new input B, and wish to test if it is better than my current input A.
I have assumed that the effect of switching from policy A -> B will have the opposite effect to switching from policy B -> A, and that either switch will lead to a step change in the reward signal after a short but unknown delay.
The population is limited to the N individuals, but I am able to switch policy from A -> B -> A -> B as many times as I wish.
My current solution:
- Split the boxes into two groups, test and control.
- Change the test group to policy B for T samples.
- Discover the mean reward over the test period for each member of each group.
- Perform a t-test on the group rewards with the null hypothesis that the average reward for the two groups is the same.
Am I able to construct a more powerful test taking advantage of the fact I can switch policy multiple times?
What field of stats/methods should I be looking at? So far I have considered looking at predictive (Granger) causality or auto-regressive models with an exogenous variable representing the policy, however neither seem to neatly fit the problem.