Confidence boundaries for an AB test Given an AB-test, I have two variants running over a certain timeperiod gathering data. I split the time period up in smaller frames and cumulative add the population for the given variant as well as the number of times the goal is reached in each group.
Afterwards I take the increase (in %) from the control to the variant group and use that as result of the experiment, which gives me a graph like this:

This smoothes out the variation over time but it is not clear whether it's a significant increase, decrease or insignificant result.
Is there a formula that can tell me at any given timeframe (which has population and number of successes for both variants) what the 95% significance boundaries are for that point in time? The aim is to verify that the positive result holds for a period of time rather than just looking at the p-value at a single moment in time.
 A: If I understand you correctly, you have two experiments, let's call them $X$ and $Y$, where at $i$-th timepoint two groups of size $n_X$ and $n_Y$ (here they are always the same for simplicity, but they can be differ at different timepoints) are assigned to two conditions and you observe $x_i$ and $y_i$ successes in both groups. So the probability of success group $X$ can be seen as a binomial random variable with $n_X$ number of trials and $\pi_X$ probability of success, that is unknown to us (same applies to $Y$).
This problem can be approached from the Bayesian perspective, using beta-binomial model, where the posterior distribution for the probability of success (by conjugacy) is a beta distribution
$$
\pi_X|x_i \sim \mathcal{B}(\alpha_X + x_i, \;\beta_X + n_X - x_i)
$$
where $\alpha_X,\beta_X$ are the parameters from the $\mathcal{B}(\alpha_X, \beta_X)$ prior distribution for $\pi_X$ (say, uniform $\alpha_X=\beta_X = 1$). If you want to update what you learned at $i$-th point, given the results at $i+1$ point, then the posterior distribution becomes
$$
\pi_X|x_i,x_{i+1} \sim \mathcal{B}(\alpha_X + x_i + x_{i+1}, \;\beta_X + 2n_X - x_i - x_{i+1})
$$
etc.
From this, you can easily obtain point estimates for $\pi_X$ (e.g. posterior mean, median, or mode) and credible intervals (e.g. taking the appropriate quantiles from the posterior distribution). You can even track the changes over time and produce a plot like the one below (here using simulated data).

Instead of plotting the individual distributions of $\pi_X$ and $\pi_Y$, you can instead plot the distribution of their differences $d = \pi_X - \pi_Y$. The distribution can be easily obtained by drawing random samples from posterior distributions of $\pi_X$ and $\pi_Y$ and then taking their differences, the empirical distribution of those differences can be used as an estimate of the posterior distribution.

Alternatively, you could look directly at the posterior predictive distributions of $X$ and $Y$, by simulating the draws from the beta-binomial distributions with appropriate parameters and then counting how often $x>y$ among the simulated values (so probability is count divided by the number of random draws).

All this, obviously, operates on the "cumulative" data, since after each new experiment you update you knowledge. That is why the intervals get narrower over time. As you asked, this gives you the intervals and enables to compare the changes in your knowledge about the parameters over time. Notice however, that you seemed to be asking about comparing the distributions at some arbitrary time window, but this approach would also take into consideration your past knowledge (what seems reasonable!).
If you assume that during some periods of the experiments there were external factors that could influence the results and this is the reason that you want to compare those periods, then you can do better then this. Instead of looking at the plots and making informal interpretations, you could use more complicated Bayesian model that includes additional covariates (e.g. weather). For this, you would define the model in terms of Bayesian logistic regression model, but this is a different story.
Since sometimes code is worth more then thousand words, I attach the R code that was used to produce the examples.
library(ggplot2)
set.seed(123)

# simulated data
n <- 100
nx <- ny <- 500
x <- rbinom(n, nx, 0.22)
y <- rbinom(n, ny, 0.21)

plot(x,type = "l", ylim = c(80, 140))
lines(y, col = "gray")

# posterior parameters for pi_X
post_x_a <- 1 + cumsum(x) 
post_x_b <- 1 + cumsum(nx - x) 

# posterior parameters for pi_Y
post_y_a <- 1 + cumsum(y) 
post_y_b <- 1 + cumsum(ny - y) 

post_df <- data.frame(
  # posterior mean and quantiles for pi_X
  x_mu = post_x_a / (post_x_a + post_x_b),
  x_ymin = qbeta(0.025, post_x_a, post_x_b),
  x_ymax = qbeta(0.975, post_x_a, post_x_b),
  # posterior mean and quantiles for pi_Y
  y_mu = post_y_a / (post_y_a + post_y_b),
  y_ymin = qbeta(0.025, post_y_a, post_y_b),
  y_ymax = qbeta(0.975, post_y_a, post_y_b)
)

ggplot(aes(y = x_mu, x = 1:n), data = post_df) +
  geom_line(color = "blue") +
  geom_ribbon(aes(
    ymin = x_ymin,
    ymax = x_ymax
  ), alpha = 0.3, fill = "lightblue") +
  geom_line(aes(y = y_mu, x = 1:n), color = "red") +
  geom_ribbon(aes(
    ymin = y_ymin,
    ymax = y_ymax
  ), alpha = 0.3, fill = "red") +
  xlab("") + ylab(expression(list(pi[X], pi[Y]))) +
  theme_minimal()

# m=50000 random draws from the posterior distribution
m <- 50000
post_x_sim <- matrix(rbeta(n*m, post_x_a, post_x_b), nrow = m, byrow = TRUE)
post_y_sim <- matrix(rbeta(n*m, post_y_a, post_y_b), nrow = m, byrow = TRUE)
# posterior distribution of d = pi_X - pi_y
post_xy_diff <- post_x_sim - post_y_sim
post_sim_df <- data.frame(
  # posterior mean
  diff_mu = colMeans(post_xy_diff),
  # posterior quantiles
  diff_ymin = apply(post_xy_diff, 2, function(x) quantile(x, probs = 0.025)),
  diff_ymax = apply(post_xy_diff, 2, function(x) quantile(x, probs = 0.975))
)

ggplot(aes(y = diff_mu, x = 1:n), data = post_sim_df) +
  geom_line(color = "darkgray") +
  geom_ribbon(aes(
    ymin = diff_ymin,
    ymax = diff_ymax
  ), alpha = 0.3, fill = "lightgray") +
  xlab("") + ylab(expression(d == pi[X] - pi[Y])) +
  theme_minimal()


library(extraDistr)

m <- 50000
post_x_sim <- matrix(rbbinom(n*m, n, post_x_a, post_x_b), nrow = m, byrow = TRUE)
post_y_sim <- matrix(rbbinom(n*m, n, post_y_a, post_y_b), nrow = m, byrow = TRUE)
post_xy_diff <- post_x_sim - post_y_sim
post_sim_df <- data.frame(
  # posterior mean
  diff_mu = colMeans(post_xy_diff),
  # posterior quantiles
  diff_ymin = apply(post_xy_diff, 2, function(x) quantile(x, probs = 0.025)),
  diff_ymax = apply(post_xy_diff, 2, function(x) quantile(x, probs = 0.975)),
  # posterior probability of pi_X > pi_Y
  diff_prob = apply(post_xy_diff, 2, function(x) mean(x > 0))
)

ggplot(aes(y = diff_prob, x = 1:n), data = post_sim_df) +
  geom_line(color = "darkgray") +
  xlab("") + ylab(expression(Pr(X > Y))) +
  theme_minimal()

