Does online A/B experimentation with simple randomisation ensure properties of a “probability sample”?

Question

High Level Problem Statement: Traditionally vendors (Optimizely, LaunchDarkly etc), who provide in-product experimentation solution to run online A/B experiment, randomise in-coming active users of the product using a simple randomisation. Does this simple randomisation ensure properties of a “probability sample”?

We are concerned if this can be argued as convenience sampling (despite randomisation). What implicit and explicit assumptions do we have here or need to argue so that this is a probability sample and NOT a convenience sample?

If this is convenience sampling, I wonder how does people (Microsoft and other companies) do traditional A/B tests using t/z-tests. A lot of vendors allow A/B experimentation and tooling for example: Optimizely, Launch Darkly etc. I wonder what is happening here.

More details: In product A/B randomised controlled experiments is executed in the following way. Usually we develop a feature and roll it out to (lets say) to 50% of users coming to the platform. The randomisation is done at the user level, when the users arrives at the product (details below).

However, as you can understand typically for designing an experiment we do pre-experiment power analysis for determining sample size and experimentation window. Now does this process make the sampling process a convenience sample?

How does the randomisation work? We use hashing based mechanics to randomise the user when we setup the feature for release using A/B test. So randomisation scheme is pre-defined when setting up the experiment using the hash(user+experiment name + config). Therefore, when user comes we use the randomisation scheme to randomise. An example method is shown here.

Answer 1

Your question conflates two types of randomization: random sampling and random assignment.

The notion of a "probability sample" refers to the former — in particular, we say a sample of $n$ units from a population of size $N$ is a "probability sample" if each of the $N$ units has a known probability of being included in the sample. This property is useful because it allows us to generalize from the sample to the population. As an example: If every unit has an equal probability of being in the sample (i.e., $p = 1/N$), then an unbiased estimator of a population mean $E[y]$ is the sample mean $\bar{y} = \frac{1}{n} \sum_{i=1}^n y_i$. (It's worth noting that in survey contexts, true probability samples don't really exist due to nonresponse.)

On the other hand, random assignment of a treatment is what occurs in a randomized experiment or A/B test. This ensures that (in expectation) the treatment and control groups are balanced on all observable and unobserved variables — including potential outcomes $Y(0)$ and $Y(1)$. As a result, simple comparisons of the treatment and control groups identify the effect of the treatment on the outcome. However, it does not ensure that the estimated treatment effects will generalize to the population of interest. For example, suppose a survey vendor did an A/B test on a sample of 18- to 24-year-olds. How they respond to an A/B test might be very different than a 65-year-old.

Most online survey vendors do not provide probability samples. They have respondents who self-select into the panels. As a result, it's impossible to know the probability that any given person will be sampled.

Answer 2

It may be a convenience sample, only in so far as you can only collect data from people who choose to interact with your product during that time and not anyone else.

That being said, it's all about interpreting the inference correctly. Is the result of the test an inference to how the change would effect all possible users? No. Its an inference to how the change would effect users who have been and will continue to use the product.

The randomization allows for a valid inference assuming a couple of things:

1. That you interpret the population from which you are sampling correctly. As you mention, there may be some hairs to split since you are not randomly sampling your user base and instead are relying on people to come to you.

2. Assuming that the test statistic has the hypothesized distribution. As an example, when the outcome is binary, the test large enough, and the baseline metric not too close to 0 or 1, then an inference via the z test is fairly good.

3. Your randomization is working and not systematically introducing confounding.

The randomization ensures that the distribution of outcomes prior to intervention is identical, thus allowing for a causal inference to be made.