I have an observational quasi-experimental study, where I try to estimate the effect of a "treatment" (participation in a programme) on a continuous outcome.
Participants (some two-thirds of all participants) are matched with non-participants on a few background characteristics. To estimate the effects (difference-in-difference), I use a multiple regression. With this method I get an estimated effect size of approximately 0,10, not significant (standard error 0.09). In the original data set I had more than 6000 treated and around 200,000 untreated in a "comparision group".
It was suggested that the actual "treatment" was too small (it ranges from 1 to 9 visits to a counselling provider) and it was suggested that the threshold for treatment should be moved up to "more than one visit".
This suggestion limits the number of treated quite badly (The number of matched controls also decreases). Defining treated as observations with more than one visit reduces the number of observations to 740. (Because of a skewed distribution in number of visits, and necessary qualifications in what constitutes a usable "treatment spell"). I am quite worried about the power of the renewed estimate. I would like to reject the suggestion, citing a further reduction in power because of the small effect size and the reduction in the sample. But how would I calculate how many observations I need to "keep the power" in this regression setting (as rebuttal)? Just calculating difference in group means does not control for secular drift or other confounders.
Please, any help appreciated. I hope I have explained my problem adequately $$ \ln Y_t = \alpha +\beta(Treated*After) + \gamma_1*Treated + \gamma_2*After +\gamma_3X $$ where $\beta$ is effect size and Treated=1 if treated (0 otherwise) After = 1 if observation period is after treatment (0 if before treatment) X a set of other explanatory variables
PS just redoing the estimations gives a new effect size of 0,064 , stderr 0,071 . More treatment, smaller average effect. well fancy that!