A/B Testing in observational studies

In the regime of observational studies, I am trying to understand if system A works better then system B.

The task of a system is to pick products from a given cart full of products. Whether a pick succeeds or not depends on the system (did it propose a good grasp) and products features that. I don't control the latter but can measure it (e.g. how tilted the product is). Each system can choose not only how to grasp a product but also which product form the given cart to grasp. In particular, the successes of consecutive picks are correlated.

For any given cart the products features may be anything, e.g. the tilt can be either close to uniformly distributed or very skewed toward 0 or 1. I have no control over it.

I am trying to find a way to obtaining statistical significancy in as tiny sample as possible. Ideally ~1000 system decisions would be enough.

In previous version of this question Peter Flom suggested to use pairing. That seems like a viable approach, as in this case one could have system A and system B work alternately and match consecutive attempts. What worries me slightly is the correlation between attempt N and N+2,..., N+M and the fact that system's A actions may actually make system's B job easier.

• What are X and T? How much data have you got? What exactly are you trying to prove? Is T the same for $x_a$ and $x_b$ (that is, are they paired somehow)? Have you considered matching? – Peter Flom - Reinstate Monica Oct 24 '19 at 10:43