How to decide time duration for synthetic control For A/B test, one can use power analysis to determine the duration of experiment to run. What about synthetic control method? Is there a way to determine the duration of experiments post intervention?
 A: There are no simple formulas, but you can use simulation to back out a test duration/horizon.
Suppose you have a candidate region $k$ for the test and an outcome $y_{kt}$ that you care about. It helps to have a maximum possible test duration. Call that $T$.
Take the time series in the candidate region $y_{kt}$. Pick a random date at least $T$ periods ago from today. Today is defined as the last period where you have mature data.
Take all data after that day and scale it up by a specified effect size $1+\Delta$ (i.e., 5 or 10%, so $\times 1.05$ or $\times 1.10$). Run the synthetic control procedure on this data, generating a forecasted counterfactual and confidence interval. If the shifted data falls outside the confidence interval after some $1 \le t' \le T$, flag that effect size as detectable for this test unit by time $t'$. Repeat this procedure many times – over different starting dates and effect sizes, and use the average empirical rejection rates of the null hypothesis to generate both a power curve and a reported Type 1 error rate.
You can be smart and start with a large effect you can reliably detect in $T$ periods and work down from there. You can do big jumps of effect size at first but then switch to smaller increments by the end. This will give you a grid of reliably detectable effects that depend on $t'$ and effect size $\Delta$.
It's a good idea to be conservative and allow some time for the effect to bake in beyond what you find above. The simulations assume the impact shows up immediately, but that is often not the case. For example, pricing experiments find bigger elasticities over time as it takes some time for users to find an alternative. A good rule of thumb is at least one or two weeks minimum if you have in-week seasonality. Make sure you have at least one week without major holidays in the test. If your business/setting is seasonal, and the test you are designing is slated to run during a peak or a lull, you need to adjust your analysis accordingly.
Make sure your historical data reflects the markets that will be available to you during the actual test. If you already have Cleveland assigned to a concurrent test that may move your test's outcome, then Cleveland can't be in your potential donor set in the simulation (or test analysis). Similarly, historical tests or shocks can contaminate historical data, so you must also prune here.
If you have multiple outcomes, you will need to repeat this simulation since the one most challenging to model will pin down the test length.
