How can you analyze how post-treatment covariates impact the outcome variable in a randomized experiment? Background
Let’s rewind time and say a company like Twitter wanted to introduce a feature that allows each user to add a banner to the top of their profile page. After creating this feature, 5k beta tester high-profile accounts were auto-enabled into the feature and then 5k other accounts organically enabled it.
Experiment
We want to design an experiment that sees if this banner impacts some engagement metric such as “Total Favorites” from each user that sees it.
To structure this experiment, we do a visitor-side randomization where visitors are exposed to the experiment when they visit any of the ~10k profiles that have the banner.
Test: Participants can see the banner if they visit a profile with a banner
Control: Participants can’t see the banner if they visit a profile with a banner
Outcome Variable: Total Favorites for each participant
What I'd like to understand

*

*How the total number of visits to an auto enabled profile with a
banner impacts Total Favorites I.e. What’s the relationship between
how many times they saw a banner and the impact on total favorites


*How the total number of visits to an organically adopted profile
with a banner impacts Total Favorites. I would want to use this to understand the
impact on favorites from buyers interacting with the organically
adopted profiles. I realize there is a bias with who has organically
adopted it, but would like to understand even with the bias.
My initial solution would be a regression model similar to below:
Total Favorites = α + β1 (enabled_high_profile_visits) + β2(organic_enabled_profile_visits) + β3 (test_treatment) + β4 (test_x_enabled_high_profile_visits) + β5 (test_x_enabled_high_profile_visits)
Definitions
enabled_high_profile_visits - total visits to profiles that had the unit auto-enabled
organic_enabled_profile_visits - total visits to profiles that organically enabled the unit
test_treatment- binary condition if you are part of the test group, and therefore have been exposed to a profile with the banner
My concern is that I will run into post-treatment bias by conditioning the regression on post-treatment variables. Is there any alternative solution, or is this not a problem in this case?
 A: I may not fully understand some of the concepts like organically adopted and auto enabled but I will try to answer the question nonetheless.  I will talk in terms related to clinical trials since that is a similar randomized setting I am familiar with.  Conditioning on post-treatment covariates can be seen as a subgroup analysis among a set of "responders" or "non-responders".  If you are interested in the treatment effect for the entire target population you should not condition on these post-treatment covariates.  If you are interested in these subpopulations, then definitely condition.
In a clinical trial setting we are often interested in inference for the entire target population.  In the trial subjects will often switch or discontinue treatment as a result of being a "responder" or "non-responder."  One option is to imagine a world where switching would not occur when defining the estimand (e.g. population-level mean).  This would amount to censoring subjects after a treatment switch and using an unverifiable missing data assumption to account for their missing values during the analysis.  Another option is to incorporate this switching into the treatment definition, i.e. a treatment policy d(A,B):=[start with treatment A; if it doesn't work switch to treatment B].  This amounts to averaging together resonders and non-responders (treatment switchers and non-switchers).  For this estimand causal inference remains at the policy level.
It sounds like you might also be worried about sampling bias $-$ that the sample of subjects in your experiment are not randomly selected in a way that is representative of the target population.  You can use inverse-probability-weighting or propensity scores to re-weight the sample in a way that you think is representative of the population.  This is typically an unverifiable assumption and should be accompanied by a sensitivity analysis.  An easier approach is to acknowledge potential bias and use a much smaller significance level (larger confidence level) when performing inference.  This doesn't remove the sampling bias, but would help to ensure the confidence interval covers the truth despite the bias.
Apologies again for not using your same terminology.  If you refine your question based on my answer I can make an attempt at improving my answering.
