I have a (large) population on which I randomly split into groups, and subjected one group to an experimental condition (call it change A) which demonstrated a significant improvement in a target variable overall. It happens that the type of the target variable is a proportion.
However, the impact of change A probably isn't the same for all subgroups. Suppose I split my population into men and women. Is change A only better for women? I only care if the difference is a change in direction, that is, I don't care if it's 15% improvement for women and a 45% improvement for men, I only care if it's 10% worse for women and 20% better for men, in which case I should only apply the change for men. The change is costless but adds a little complexity to apply it only conditionally so if it's better for one subgroup and shows no significant difference for another subgroup, I'd rather apply the change across the board. I am only interested in whether the change is significantly positive for one group and significantly negative in another.
The division into subgroups is somewhat arbitrary, men and women are presumably something on the order of a 50/50 split, but some of the segmentations I could make would be orders of magnitude smaller than the original population, like product brand usage or county or something like that, where the difference might not that significant for individual segments due the size of the subpopulation. Would it be a good idea to keep track of effectiveness of change A, for individual counties, and only enable it for counties where the change was positive? In some counties the change was better, in some it was worse. How can I tell whether the differences are not due to random variation?