1
$\begingroup$

Context

Let us say we are interested in the Average Treatment Effect (ATE) as an estimand. Following the potential outcomes framework, we define it as: $$\frac{1}{N} \sum_{1}^{N}(Y_i^0 - Y_i^1)$$

where $Y_i^a$ is the potential outcome for subject $i$ under condition $A = a$, $A$ is a treatment assignment variable that takes values $0$ or $1$ for control and treatment, respectively, and $N$ is the number of subjects in the population.

Let us define two working models for the causal structure:

  • DAG 1, with a fork on $X1$ and on $X2$:

    • $A \leftarrow X1 \rightarrow Y$,
    • $A \leftarrow X2 \rightarrow Y$ and
    • $A \rightarrow Y$
  • DAG 2, with mediator $X1$ and a fork on $X2$:

    • $A \rightarrow X1 \rightarrow Y$,
    • $A \leftarrow X2 \rightarrow Y$ and
    • $A \rightarrow Y$

Clearly, in DAG 1 both $X1$ and $X2$ are confounders, whereas in DAG 2 only $X2$ confounds the relationship between $A$ and $Y$. Using a regression model as estimation strategy (OLS), we arrive to two models that depend on our causal assumptions (adding a subscript to indicate each one):

  • $Y_i = \beta_{10} + \beta_{11} A_i + \beta_{12} X1_i + \beta_{13} X2_i + \epsilon_i$, where we use our estimate of $\hat{\beta}_{11}$ as an estimate for the ATE.
  • $Y_i = \beta_{20} + \beta_{21} A_i + \beta_{22} X2_i + \epsilon_i$, where we use our estimate of $\hat{\beta}_{21}$ as an estimate for the ATE.

Question

Considering that an estimand is a "quantity to be estimated in a statistical analysis", which of the following assertions is false and why?:

  • $\beta_{11}$ and $\beta_{21}$ are different quantities being estimated and thus are different estimands
  • ATE is the only estimand, with $\hat{\beta}_{11}$ and $\hat{\beta}_{21}$ being two estimates for it

(I know that the relationship between an estimand and its estimator can be fully arbitrary. For instance I could roll a die and have the result be the estimator for the average height of people in my country. It would be a useless estimator, but an estimator nonetheless. My question is about whether an estimand is uniquely defined in such context or if it is decidedly ambiguous. The motivation is thinking about its consequences in model averaging, for example)

$\endgroup$
11
  • $\begingroup$ I believe it is answered in detail in the thread linked as a duplicate. TL;DR estimand is the quantity you estimate, estimator is the method used for estimating it, and estimate is the result produced by the estimator. $\endgroup$
    – Tim
    Nov 9 '21 at 14:07
  • $\begingroup$ I think the duplicate does not answer this question. The crux of the issue is exactly that the definition of estimand as 'the quantity you estimate' is sufficiently vague, that in some frameworks this issue has to be compensated with further distinction between a theoretical estimand (like the ATE) and an empirical estimand (see journals.sagepub.com/doi/abs/10.1177/00031224211004187). I don't think this approach is standard and hence my question. On reading the duplicate maybe what is confusing is calling the beta hats estimators instead of estimates, which will I edit. $\endgroup$
    – Kuku
    Nov 9 '21 at 15:54
  • $\begingroup$ I'm not sure what is your question? The definition is broad since it is a very general concept. Regarding question from your question: every parameter you estimate is estimand. $\endgroup$
    – Tim
    Nov 9 '21 at 16:08
  • 1
    $\begingroup$ It is answered since it defines the concepts. Estimand is the thing you estimate. Same as divisor is the thing you divide by. $\endgroup$
    – Tim
    Nov 9 '21 at 16:17
  • 1
    $\begingroup$ They both estimate the parameters of the regression model. You can have multiple estimators for the same thing. As you noticed, anything could be an estimator and you can have multiple possible estimators for the same thing and this is a common case in statistics to have more than one possible estimator (e.g. you could estimate regression using OLS, MLE, or Bayesian approach). $\endgroup$
    – Tim
    Nov 9 '21 at 18:26
2
$\begingroup$

The ATE is an estimand involving unseen potential outcomes and is defined at $E[Y^1-Y^0]$, where $Y^1$ and $Y^0$ are the potential outcomes under treatment and control. Under the main causal assumptions, the ATE is equal to $E[E[Y|A = 1, V]-E[Y|A=0, V]]$, where $V$ is a valid adjustment set. Let's call $E[E[Y|A = 1, V]-E[Y|A=0, V]]$ the average marginal effect (AME), which doesn't have a causal interpretation except when the assumptions that make the ATE equal to the AME are satisfied. The AME is also an estimand, but it doesn't require specific causal assumptions to be true to estimate it. It is possible there are multiple sets $V$ that make the AME with respect to $V$ equal to the ATE.

When a model is parameterized in a certain way, it is possible for a parameter in that model to correspond to the AME under certain assumptions that link the model parameter to the estimand.

Consider the following estimands:

  • $AME_{12} = E[E[Y|A = 1, X_1, X_2]-E[Y|A=0, X_1, X_2]]$
  • $AME_2 = E[E[Y|A = 1, X_2]-E[Y|A=0, X_2]]$

Under DAG 1, $AME_{12}$ is equal to the ATE, and $AME_2$ is a confounded association between $A$ and $Y$. Under DAG 2, $AME_2$ is equal to the ATE, and $AME_{12}$ is the direct effect of $A$ on $Y$ not through $X_1$.

Consider that the true outcome model is linear in the covariates and treatment and that there is no interaction between the treatment and covariates (i.e., so that your first model perfectly describes the data-generating process, which is consistent with both DAG 1 and DAG 2). Under this assumption, in your first model, $\beta_{11}$ is equal to $AME_{12}$, and in your second model, $\beta_{21}$ is equal to $AME_2$.

So, under certain assumptions, a $\beta$ is equal to an AME, and under additional assumptions, the AME is equal to the ATE. So what quantity does $\hat{\beta}_{21}$ in an OLS regression correspond to in your second model estimate? It estimates $\beta_{21}$. How you interpret that with respect to an estimand depends on the assumptions you make that link $\beta_{21}$ to the estimand you desire.

It is possible to estimate the AME using a different method, e.g., inverse probability weighting (IPW). IPW does not involve specifying a regression model for the outcome; therefore, the IPW estimand does not necessarily correspond to $\beta$ in any regression model. In this way, even if we aren't willing to make the assumptions that would link $\beta$ in some regression model to the AME, we can still use IPW to estimate the AME. This is important because we can describe the AME as an estimand separate from $\beta$, which hopefully clarifies that $\beta$ and the AME are not the same estimand except when specific assumptions link them. Similarly, IPW does not target $\beta$ except when $\beta$ is equal to the AME by virtue of the linking assumptions.

Let's wrap it up: the ATE, $AME_{12}$, $AME_2$, $\beta_{12}$ and $\beta_2$ are all potential estimands. The OLS estimator of $\hat{\beta}_{21}$ is generally unbiased for $\beta_{21}$. Under certain assumptions, $\beta_{21}$ may be equal to $AME_2$. Under additional assumptions, $AME_2$ might be equal to the ATE. If these assumptions are all true, then you can say $\hat{\beta}_{21}$ is an unbiased estimator of the ATE. But, again, whether that is true depends on the assumptions linking each quantity to the next; some of those assumptions are encoded in the DAG and others in the form of the outcome model.

$\endgroup$
13
  • $\begingroup$ Thank you very much for your response, Noah. It targets exactly my issue in this case, in how the definition of an estimand is theory-mediated (i.e. assumption-mediated). So the answer to the alternative given would be that there is no unequivocal way to choose between both options, no? How does this relate to model averaging then? It seems then defensible to state, under certain assumptions, that adjusting for different covariate sets is changing the estimand, even if the goal is to estimate the ATE. Then we would be averaging different quantities assuming they reflect one thing, no? $\endgroup$
    – Kuku
    Nov 10 '21 at 12:11
  • $\begingroup$ Multiple adjustment sets $V$ can yields AMEs that all correspond to a single ATE; this is when model averaging is used. Model averaging/specification search is invalid without a causal theory that determines which variables form a valid adjustment set. You don't just average any model that is possible to be fit; you need to average models that all have the potential to target the same estimand. $\endgroup$
    – Noah
    Nov 12 '21 at 15:42
  • $\begingroup$ How can different conditional means correspond to the same thing? I am aware you don't average over any model, but if there is uncertainty as to which adjustment set is valid in a PDAG set, you are holding different sets of assumptions at the same time. And then your use of the term 'potential' is at the crux of my issue in this question. At least, without a distinction of different types or orders of estimands, the idea that the estimand here is the ATE and not the AME's through the OLS coefficients seems a matter of wishful thinking. And it surprises me if this is the general convention. $\endgroup$
    – Kuku
    Nov 13 '21 at 10:56
  • 1
    $\begingroup$ I want to estimate the ATE. Therefore it is my estimand. Assumptions tell me the ATE is equal to an AME. So the AME is also my estimand. Further assumptions tell me the AME is equal to a regression coefficient. So the coefficient is also my estimand. If any of those assumptions are not true, or if you take an estimation approach that doesn't rely on them, then what is and isn't your estimand may differ. For example, if I use IPW to estimate the AME, I'm not relying on the assumptions that link the AME to $\beta$, so $\beta$ is not my estimand, but the AME and ATE are. $\endgroup$
    – Noah
    Nov 13 '21 at 17:38
  • 1
    $\begingroup$ That is exactly right. Of course typically the purpose of estimating $\beta$ is with the hope that $\beta$ has a useful (i.e., causal) interpretation. Even if you have the right adjustment set, $\beta$ might not correspond to anything useful (e.g., the AME) depending on how the model is parameterized. For example, when $X$ and $X \times Z$ are predictors in a model, the coefficient on $X$ is generally not equal to the AME. That doesn't mean it isn't possible to construct an estimator of the AME from the model, but it just means $\beta$ isn't that estimator. $\endgroup$
    – Noah
    Nov 15 '21 at 16:53
0
$\begingroup$

I prefer to view it according to your second bullet point. You have clearly identified the estimand of interest at the very top. The two different models can be seen as sensitivity analyses, each one relying on an unverifiable assumption. This is related to the ICH E9 addendum on estimands for clinical trials. In that setting we would define a particular population-level treatment effect of interest as the estimand. Whether we make a missing at random given treatment assumption, or missing at random given treatment and age assumption, or a missing not at random assumption for post-baseline confounding, these are all different missing data assumptions one could use in pursuit of the originally defined estimand.

$\endgroup$
3
  • $\begingroup$ What would be needed to suggest that any kind of estimate for the ATE is not a sensitivity analysis, but the main analysis? In other words, what alternative estimate would not depend on unverifiable assumptions? $\endgroup$
    – Kuku
    Nov 13 '21 at 11:00
  • $\begingroup$ In practice there is no one who can tell us the "true" model for how confounding was introduced, we can only posit one. Any one model can be deemed by the experimenter as primary and the rest as sensitivity. This is done in clinical drug development for the purposes of decision making so that the decision maker does not just pick the model that produces the results that they were hoping to see. The experimenter does not necessarily have to label one model as primary and the rest as sensitivity $-$ each model can be a sensitivity for the others. $\endgroup$ Nov 13 '21 at 14:24
  • $\begingroup$ Ideally all models will produce conclusions that point in the same direction. $\endgroup$ Nov 13 '21 at 14:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.