In the regime of observational studies, I am trying to understand if system A works better then system B.

The task of a system is to pick products from a given cart full of products. Whether a pick succeeds or not depends on the system (did it propose a good grasp) and products features that. I don't control the latter but can measure it (e.g. how tilted the product is). Each system can choose not only how to grasp a product but also which product form the given cart to grasp. In particular, the successes of consecutive picks are correlated.

For any given cart the products features may be anything, e.g. the tilt can be either close to uniformly distributed or very skewed toward 0 or 1. I have no control over it.

I am trying to find a way to obtaining statistical significancy in as tiny sample as possible. Ideally ~1000 system decisions would be enough.

In previous version of this question Peter Flom suggested to use pairing. That seems like a viable approach, as in this case one could have system A and system B work alternately and match consecutive attempts. What worries me slightly is the correlation between attempt N and N+2,..., N+M and the fact that system's A actions may actually make system's B job easier.

  • $\begingroup$ What are X and T? How much data have you got? What exactly are you trying to prove? Is T the same for $x_a$ and $x_b$ (that is, are they paired somehow)? Have you considered matching? $\endgroup$
    – Peter Flom
    Oct 24, 2019 at 10:43

1 Answer 1


There are at least two broad sorts of ways to deal with this.

One is some sort of matching method. There are a bunch of these, which one is right depends on your exact situation. If the two samples are the same size (at least roughly) then 1-1 matching might work. Some choices then are greedy matching and optimal matching. There's lots written about both (and other methods). If you have a specific question about this, you could ask it.

The other would be to use in a regression (which sort of regression would depend on the qualities of X - is it continuous, is it bounded, is it a count, etc - and having group and T and group as independent variables.

  • $\begingroup$ Thank you for your answer! I updated the question to be more precise. $\endgroup$
    – user1414
    Oct 25, 2019 at 5:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.