I am trying to measure the effect of treatment after some time on a group of subjects who were selected for treatment because they were the highest (top 1200) ranked subjects during a single period marking the start of the experiment, the hope for the treatment being that they will have lower ranks in the future than they otherwise would. (Note: I did not design the experiment.)
No controls were set aside from this selection, making this essentially an observational study rather than a randomized experiment for all intents and purposes. The only candidates for controls are those subjects who were not in the top 1200 that period, meaning all other subjects in previous time periods where no experiment was being conducted. However, there are obviously a few confounding factors:
- Because they were selected based on top rank, there are going to be regression-to-the-mean effects that must be accounted for;
- Any trend in the metric of interest might have a trend that would make matching on raw values problematic.
The initial plan of action was as follows:
- All subjects of similar rank from prior time periods (who are also test subjects) are candidates for control. This is a minimal requirement to deal with regression effects.
- Any trend in the metric should be found and the series should be detrended, i.e. work with the residuals, rather than the raw data. Use only data from periods prior to the start of the experiment.
- Match test subjects to control candidates based on similarity over some selection of points that correspond to a "pre" treatment period, e.g. t-9, t-6, t-3, t0, using a caliper of, say, 0.25 SD. This is to establish that they are sufficiently similar prior to the start of treatment.
- Use a Wilcoxon test to compare treatment subjects with the averaged matched controls for each subject on some "post" period change, e.g. the change from t0 to t+6.
Problems I'm running into are that the trend is not a simple linear trend, first of all. I'm also at a loss over which data should be included in my trend determination. Also, since the treatment group is expected to have some impact on the trend post-intervention, and since there are few examples of people of similar rank during that time frame, it's difficult to determine if the trend selected is holding well as it's projected forward. On clue to the fact that things aren't working out is that I tested the protocol using only older data and a dummy test group. This experiment should result in something not significantly different than zero, but I get results that are have a significant difference. This casts doubt on all my actual results.
There also might be holes in the protocol I've described that need improving. I could be going about things all wrong and I'm beginning to think that's the case, that it's impossible to approach this way. Does anyone have any suggestions on how to better go about tackling this problem? I'm open to changing how I'm doing anything here, even totally different approaches to the problem. At the end of the day I need a statistically defensible method of saying there is or isn't a treatment effect and some way of talking about its magnitude.
To summarize: I have a longitudinal data set tracking an irregularly trending metric. At some point along the way, some number of subjects were selected for being the current leaders in the metric and treated. Then more data was collected on everyone for some more time. How can I best determine the presence and magnitude of any treatment effect (90-95% confidence interval with p-value ideally).
FWIW, I'm doing the analysis in R, if anyone wants to suggest helpful methods or packages. General advice is also welcome.