0
$\begingroup$

I'm currently designing an A/B-Test on a geo level, where the test- and controlgroup will be made up of various (US) States.

Each state has a few specific characteristics (e.g. mean income, urbanization, etc) and a sample size (absolute and relative). I would like to find an elegant way to define my control- and test-group and keep the covariates as comparable as possible.

Ideally I would also be able to control the size of the groups (say, 25% percent of the population size each) and have the groups defined in a "bottom up" way, that can be checked afterwards.

I've looked into R packages such as MatchIt or cobalt, which seem great and generally appropriate, though they seem by default to focus more on matching "individual cases" rather than "aggregated entities".

I'm looking for general support how to tackle this question as well as software recommendations. :)

$\endgroup$
1
  • 1
    $\begingroup$ It's a bit unclear what you're asking. Are you trying to find a subset of the 50 states to run a state-level experiment on? Or are you trying to find a subset of the population of the US to run an experiment on? If the latter, do you need them to be within a small subset of states? Is the treatment going to be constant across residents in each state? Is the design meant to be close to a multi-site trial or a cluster-randomized trial? Please provide more detail because it is hard to know what you're actually looking for and why existing solutions don't work. $\endgroup$
    – Noah
    Mar 29, 2021 at 18:14

1 Answer 1

1
$\begingroup$

Note: Recommendations of general approach and/or software are usually off-topic. Though as publicly available work on specifically A/B Geo-experiments region matching (as opposed to general propensity score matching between entities) is so few and far between, I think the question is objectively answerable.


Barajas et al. [1] has described an algorithm to find good matching pair of markets (i.e. 1 control state vs 1 treatment state in your case). Au [2] described how they find near-optimal matching sets of geo-regions (i.e. multiple control states vs multiple treatment states) using a hill climbing algorithm.

Both work utilises Bayesian structural times series to capture the pre-experiment relationship between the treatment and control group metric(s), and produce a counterfactual during the actual experiment period that can be compared to what the treatment group actually attained. One popular R package for this line of work is CausalImpact (see this doc).


[1] Barajas, Zidar, and Bay (2020) Advertising Incrementality Measurement using Controlled Geo-Experiments: The Universal App Campaign Case Study. In: AdKDD 2020 Workshop (in conjunction with KDD '20). http://papers.adkdd.org/2020/papers/adkdd20-barajas-advertising.pdf

[2] Au (2018) A Time-Based Regression Matched Markets Approach for Designing Geo Experiments. Technical report. https://storage.googleapis.com/pub-tools-public-publication-data/pdf/b1976d70ccf7119f2193ece2d3d378d5dd0dd7be.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.