The problem: I have N (~500) black boxes which each receive an input and output a noisy reward signal. The reward signal is non-stationary and heteroscedastic, but can be assumed stationary over short time samples. The population as a whole experience similar trends, but individuals are capable of wildly different patterns.

I have a new input B, and wish to test if it is better than my current input A.

I have assumed that the effect of switching from policy A -> B will have the opposite effect to switching from policy B -> A, and that either switch will lead to a step change in the reward signal after a short but unknown delay.

The population is limited to the N individuals, but I am able to switch policy from A -> B -> A -> B as many times as I wish.

My current solution:

  1. Split the boxes into two groups, test and control.
  2. Change the test group to policy B for T samples.
  3. Discover the mean reward over the test period for each member of each group.
  4. Perform a t-test on the group rewards with the null hypothesis that the average reward for the two groups is the same.

Am I able to construct a more powerful test taking advantage of the fact I can switch policy multiple times?

What field of stats/methods should I be looking at? So far I have considered looking at predictive (Granger) causality or auto-regressive models with an exogenous variable representing the policy, however neither seem to neatly fit the problem.

  • $\begingroup$ By "more powerful," do you mean statistical power? There are many fields of stats/methods that might help you answer your question. A t-test might work for you. In that case, make sure to include effect sizes like Cohen's d. A multiple baseline design might work. Interrupted time series autoregressive models might work, too. $\endgroup$ – Jay Schyler Raadt Oct 15 '18 at 17:38
  • $\begingroup$ Indeed I do mean more powerful, we're looking for a potentially very small effect (~1% change) in the result signal. Thank you for the pointer towards Cohen's d and effect sizes - I will definitely be include these in my analysis. One quick follow up question - apart from a slight violation of sample independence would I be able to effectively double the population by going A->B->A->B and combining the results of both tests together? $\endgroup$ – Clibbon Oct 16 '18 at 9:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.