Significance testing when the treatment group was only partially treated (an unknown set of individuals did not receive the treatment)

Question

I have a somewhat messy experiment that has already been conducted with large test and control groups, and have measured the response variables (sales / individual, etc). A known percentage of the 'test group' (supposed to be exposed to a treatment) did not actually get exposed to the treatment, and I have no way of separating them in the data. I can see whether someone was 'supposed to receive a treatment' and 'definitely did not' (control).

I can directly observe:

• Z = Intended test group (composed of X and Y) - only some received treatment
• C = Control group
• w = proportion of X that makes up Z (but cannot tie to individual records) - e.g. 60% of the intended test group actually recieved the treatment.

I cannot observe:

• X = 'Actual test group' that recieved treatment
• Y = People who were supposed to recieve treatment but did not.

I would like to be able to make statements of significance about whether X differs from C (e.g. with 2-sample t-test)

Intuitively, Y 'should' look like C (they both represent sales/customer for customers who have not received the treatment, though they have different/known sizes), and I want to subtract it out to get an estimate of X I can use in a t-test or similar. Subtracting the means proportionally I think should be fine, but I'm concerned that doing the same for the variance is not correct.

I know 'when random variables subtract, the variances add', but wonder if that will screw up my t-test since I only have the sample variances and not the population parameters.

How should I be trying to make this adjustment?

Answer 1

I disagree with the other answer by @mkt that no inference is possible, but yes, inference, like hypothesis testing, might be difficult. But there are some possibilities.

In the treatment group, where only about 60% of the subjects/items actually received treatment, we might model the distribution of the outcome as a mixture distribution, of the form $$ 0.6 \cdot \text{distribution when treatment given}+0.4 \cdot\text{distribution when treatment not given} $$ Using regression with such a model is called mixture regression (search this site, many posts!) and there are for instance some R packages, like flexmix and fpc. But our situation is simpler, since the mixing probabilities is given, known, and not to be estimated. So we can just write down a model, find the loglikelihood function, and use mle (maximum likelihood estimation.)

Since little context is given, we will use a very simple model as illustration. We assume independence and normal distributions. In the control group (and the non-treated in the treatment group) we have $Y_i \sim \mathcal{N}(\mu,\sigma^2)$ while among the treated we have $Y_i\sim\mathcal{N}(\mu+\Delta,\sigma^2)$. So in the "treatment group" we have $Y_i \sim w\cdot\mathcal{N}(\mu+\Delta,\sigma^2)+(1-w)\cdot\mathcal{N}(\mu,\sigma^2)$. For your data you told us $w=0.6$. The same principles could be used with other error distributions, covariables, etc. The interest or focus parameter is $\Delta$, so we will ultimately use the profile likelihood for $\Delta$ to construct a confidence interval.

With simulated data I construct this model in R, and the resulting confidence interval based on the profile likelihood is below:

Note that in this proposed model we have assumed constant variance. This might be a critical assumption (as it always is, but maybe more so here.) This is because, in the "treatment" group, a higher empirical variance could be because of either a large $\Delta$, or because the variance actually is larger than in the control group (maybe because the treatment, somehow, also increases variance.) So this model should be taken as tentative, it would be wise to investigate it further before use. I tried to search for papers on this issue, could not find any.

The R code used:

    n <- 100 # "treatment" sample size
    m <- 100 # control sample size
    mu <- 10; sigma <- 3; delta <- 0.8
    w <- 0.6 # fraction of treatment data really treated
    # Simulating some data:
    set.seed(7*11*13) # My public seed
    Y <- rnorm(n+m, c(rep(mu, m), rep(mu+delta, w*n), 
            rep(mu, (1-w)*n)), sigma)
    G <- c(rep(0L, m), rep(1L, n)) # 1 is treatment group
    mydf <- data.frame(Y=Y, G=G); rm(Y, G)
    
    # loglikelihood function:
    
    loglik0 <- function(mu, delta, sigma) {
        loglik <- sum(ifelse(mydf$G==0L, dnorm(mydf$Y, mu, sigma, 
            log=TRUE), log(w*dnorm(mydf$Y, mu+delta, sigma) + 
        (1-w)*dnorm(mydf$Y, mu, sigma))))
        -loglik  # we use bbmle::mle2 which requires the 
                 # negative loglik
    }
    
    library(bbmle)# on CRAN
    mod <- bbmle::mle2(loglik0, start=list(mu=8, delta=0, 
                 sigma=2))
    
    mod.prof <- bbmle::profile(mod, which=2)
    
    confint(mod.prof)
        2.5 %    97.5 % 
    -0.571204  1.704091 
    
    plot(mod.prof)

Answer 2

EDIT: kjetil b halvorsen's new answer has persuaded me that my answer below is incorrect.

---

+1, this is an interesting question

To summarize: the goal is to try to compare people who did receive treatment ($X$) with those who were controls ($C$). However, $X$ is a subset of the population supposed to receive treatment ($Z$), and we only know that $X$ is $w$% (60% here) of $Z$, with the remainder (100 - $w$)% being called $Y$.

---

Unfortunately, I do not think there is any legitimate way to make a strong statistical claim from this data. Therefore, I think estimating statistical significance is out of the question.

However, it might be possible to do a useful exploratory analysis with this dataset instead.

I would start by examining $Z$, the people supposed to have received treatment. If you run a clustering algorithm on this, you might find that the data falls into two natural groups with a ~60/40 split, as expected. These would correspond to the subgroups that did receive treatment ($X$) and those that were supposed to but did not ($Y$). You could then calculate the means of the 3 groups - $X$, $Y$ and $C$ - and with caveats, make some weak statements about possible differences between them. You could also exclude $Y$ from further consideration, or add those points to $C$, but I think keeping them as a separate category is less likely to mislead.

Assuming the above steps work, could you then do a hypothesis test on them? No - because the results are going to be biased to an unknown degree by the uncertainties in the clustering step. Points belonging to $X$ but that are far from its mean/median/centroid (depending on clustering approach) may be grouped with $Y$, and vice versa. This will have the effect of reducing group variances and inflating differences between the groups. Despite these limitations, I think that this approach could provide useful information about the differences between groups.

A few related thoughts:

1. A clean clustering result like the one I described might seem very unlikely - clusters could be the wrong proportion (i.e. not corresponding well to $w$% and (1 - $w$)%, or more than two clusters might be identified in the data. If either of these happen, I might try a different cluster approach, but my (already low) confidence in this procedure would reduce substantially. But if the treatment has a strong effect, which may be expected based on prior knowledge and experimental design, two distinct clusters of approximately correct proportions are an entirely plausible outcome.
2. Knowing $w$ is crucial for getting some idea of how well the clustering step has worked.

3. $w$ being quite different from 50% is helpful knowing which clusters are $X$ and $Y$. However, even if $w$ was exactly 50% and the two groups therefore identical in size, comparing the two clusters against $C$ would help in distinguishing $X$ from $Y$. If one cluster's outcomes are much more similar to those of $C$, it is likely to be $Y$.

4. I'm setting aside discussion of what type of clustering approach to use because there are a huge range of possibilities, and which one works best depends strongly on details that we do not know.