A/B testing segments post-analysis have the data of an A/B testing experiment. We can assume that the power analysis has been done correctly, so both sets, control and treatment had the necessary samples to find stat sig results.
The thing is that within the control/treatment sets, exist 3 segments, which I would like to analyze and detect if any result within each segment is stat sig, in this case, we can't assume that power analysis was done for each one of the segments.
How can I analyze the whole experiments and the sub-segments?
My idea until now is:
For the whole dataset: perform a stat sig test
For the segments: perform a stat sig test, but adjust the alpha to a lower value, kind of a Bonferoni adjustment.
Any ideas on other methods to evaluate the segments?
 A: As you suspect, there's nothing you can do to increase the power of hypothesis tests on the segments after the fact. You're stuck with the power you have. The best thing you can do is control the Type 1 error in an efficient way so you don't lose power.
Remember that any time you perform a hypothesis test, you risk making a Type I error. In your suggested approach, you control the Type I error in the whole dataset and the segments separately. It is possible to incorrectly reject the null hypothesis on the whole data and incorrectly reject the null of at least one of the segments. In more technical terms, you are controlling the experimentwise error rate but not the family-wise error rate.
I have two suggestions.

*

*I would gate your hypothesis tests of the segments behind the test of the entire population. This means you first perform a hypothesis test on the entire population and only continue to perform tests on the segments if your test on the entire population is significant.


*The Bonferonni procedure is very inefficient. Several other procedures are more powerful and still maintain the same Type 1 error rate. I recommend the Hochberg Procedure. It is explained in the link I provided. The Hommel procedure is more powerful than Hochberg, but it is more complicated and fewer people are familiar with it. Both procedures can be implemented in R.
To summarize, perform a hypothesis test on the whole population, then, if you reject that hypothesis, move on to testing the segments using a Hochberg procedure.
A: Assuming the outcome is binary (though the following will work for any outcome), we can:

*

*Model an interaction between treatment and segment, and

*Test for an effect of the treatment through a likelihood ratio test.

Let's assume we have designed this experiment with the hypothesis that there is an effect and it varies by group.  Doing otherwise (looking at the data first and then forming a hypothesis) would be an invalid approach. Assume we run the experiment and our data look like
    txt segment     y     N
  <int> <fct>   <int> <int>
1     0 1         143  1669
2     0 2         374  1596
3     0 3         485  1735
4     1 1         211  1701
5     1 2         645  1673
6     1 3         241  1626

Using R, we can fit a model which allows treatment effect to vary by group using
model = glm(cbind(y, N-y) ~ txt*segment, data = d, family = binomial())

The summary of the model can be examined to determine the estimated effects of each group and treatment.  I find it easier to plot the predicted probabilities.  For the data I presented, such a plot would look like...

It certainly looks like there is an effect of treatment and that effect varies by group. Let's test that with a likelihood ratio test.  First, we create a null model which adjusts for group in an additive fashion.  Then, we pass our model and our null model to the anova function.
# Create a null model where there is no effect of group
null_txt_model = glm(cbind(y, N-y) ~ txt + segment, data = d, family = binomial())
anova(null_txt_model, model, test='LRT')

Analysis of Deviance Table

Model 1: cbind(y, N - y) ~ txt + segment
Model 2: cbind(y, N - y) ~ txt * segment
  Resid. Df Resid. Dev Df Deviance  Pr(>Chi)    
1         2     182.91                          
2         0       0.00  2   182.91 < 2.2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


Our model leads to a reduction in deviance large enough to reject the null hypothesis.  We would conclude that the treatment effect does depend on group.  Had we failed to reject this, we could not rule out a purely additive effect of group.
The problem with this analysis is that if you've powered your experiment to detect an effect of the treatment, you've likely underpowered it for this sort of analysis.  Nothing you can do about it at this point.
