Causal inference where potential outcome is somehow "violated"?

Question

The fundamental problem of causal inference says that only one potential outcome is observed for each unit.

What happens if both outcomes from control and treatment can be observed? Can we still make use of analysis tools like causal trees to understand heterogeneous treatment effects?

As a concrete example, suppose we are an online search engine and want to better understand how to serve ads. Each time a user enters a search query (request), we pick an ad from a collection of $N$ ads and show it to the user. For each of these $N$ ads, there are 2 versions, one with an image and one without. We randomize users into a control group and a treatment group (same distribution of users in each group), where users in the control group will see ads without an image and users in the treatment group will see ads with an image. By the end of the experiment, for each ad we record the total number of clicks by users in the control group as well as in the treatment group.

In this particular case, each ad is an experimental unit, and we are able to observe outcomes from both the treatment group and the control group. We want to understand if we should add an image to the ads or not.

In addition, each ad has its feature (e.g., associated company, promoted product, etc.), and we want to understand which group of ads will benefit the most from the addition of images. My question is, in such a case, can we follow the routines in treatment effect analysis? If not, what's a more suitable framework?

Answer 1

I agree that there is some confusion about the "unit" of analysis here. It's neither the ad nor the viewer, though; it's the instance of showing an ad to a viewer. And there is only one potential outcome observed because that instance can only either have an image or not. Because you randomly assigned, you don't have to worry about confounding, which is nice, but that's not the same thing as having both potential outcomes for each unit.

It happens to be that instances are nested within specific ads, but the specific ad is a characteristic of the instance.

You can estimate a number of quantities from this design. You can estimate the average treatment effect of pictures by simply comparing the outcomes between the instance with pictures and the instance without. You should additionally control for the specific ad and any user-level qualities as well to increase the precision of your estimate and improve estimation of the standard error. To do this, you could fit a fixed effects or random effects model with the treatment as the primary predictor and the specific ad as the fixed or random effect grouping variable, e.g., Y ~ treat + (1|ad) if using lme4 for random effects or Y ~ treat | ad if using fixest for fixed effects (the results should be similar).

You can also estimate the ad-specific treatment effect, which is the effect of showing a picture for a specific ad. This is no different from a subgroup average treatment effect; it is essentially interpreted as if you only had one ad but showed it several times with and without the picture. You can estimate these effects in a single model using the following syntax in R: Y ~ ad/treat - 1 in lm(). This gives you a treatment effect for each ad. This would only make sense if you had many instances of each ad with both a picture and no picture.

If you are interested not in specific ads but perhaps the effect of showing the picture for other user- or ad-level characteristics, you can estimate heterogeneous treatment effects using causal trees in the standard way; you just would not include the specific ad as a predictor if you were looking at ad-level characteristics. If you had specific hypotheses, you could also test them in the models above by including the predictor of interest in an interaction with treatment.

Answer 2

You misunderstood the definition of unit there. One unit, individual, can not be in the control group and the treatment group at the same time. You can only observe the effect of ONE intervention on an individual, at a given time. The two types of ads in your exemple are the treatment, say A and B, and the visitors to your website are the individuals, the "patients".

In experiments we see all the time what happens to participants of the study in all groups, but we can not see what happened to Bob when he took the pill and when he didn't take it, ceteris paribus(keeping all the rest constant).

This is not an assumption, so you can not violate it. It's a problem, and one way of solving it, maybe, is traveling in time :-) A bit tricky

Answer 3

I would set up your data as an ad-level crosssection, where:

• Each row is a distinct ad
• Ad characteristics are the other columns.
• The outcome column is the treatment-control difference in the two clicks per impression rates.
• I would also include the number of impressions that the difference is calculated from as a column, in case some ads appear more frequently than others.

You can fit a model of the effect as a function of the ad-level variables, possibly using the number of impressions to weight the data. If you have few ad characteristics relative to ads, linear regression or a non-parametric model will work. If you have many covariates relative to the number of ads, the lasso modified for inference is a good first start:

Belloni, Alexandre, Victor Chernozhukov, and Christian Hansen. 2014. "High-Dimensional Methods and Inference on Structural and Treatment Effects." Journal of Economic Perspectives, 28 (2): 29-50. https://www.aeaweb.org/articles?id=10.1257/jep.28.2.29

This modification is similar in spirit to the causal trees approach you mentioned.

Answer 4

The fundamental problem of causal inference defines the impossibility of associating causality with a correlation; in other words, correlation does not prove causality. This problem can be understood from two perspectives: experimental and statistical. The experimental approach suggests that this problem arises from the impossibility of observing an event both in the presence and absence of a hypothesis simultaneously. The statistical approach, on the other hand, suggests that this problem stems from the error of treating tested hypotheses as independent of each other.

Modern statistics tends to place greater emphasis on the statistical approach, as it, unlike the experimental approach, also provides a path to solving the problem. Indeed, when testing many hypotheses, a composite hypothesis is constructed that tends to cover the entire solution space. Consequently, the composite hypothesis can fit any data series, thereby generating a correlation that does not imply causality.

Furthermore, the probability that the correlation is random is equal to the probability of obtaining the same result by generating an equivalent number of random hypotheses. Regarding this topic, we will see that the key point, in calculating this probability value, is to consider hypotheses as dependent on all other previously tested hypotheses.

Considering the hypothesis as non-independent has fundamental implications in statistical analysis. In fact, every random action we take is not only useless but will increase the probability of a random correlation. For this reason, in the following article “Statistics the Science of Awareness”, we highlight the importance of acting consciously in statistics.

Moreover, calculating the probability that the correlation is random is only possible if all prior attempts are known. In practice, calculating this probability is very difficult because not only do we need to consider all our attempts, but we also need to consider the attempts made by everyone else performing the same task. In fact, a group of people belonging to a research network all having the

same reputation and all working on the same problem can be considered with a single person who performs all the attempts made. From a practical point of view, we are almost always in the situation where this parameter is underestimated because it is very difficult to know all the hypotheses tested. Consequently, the calculation of the probability that a correlation is casual becomes something relative that depends on the information we have.