Suppose that I have a regression model

\begin{align} Y = X\beta_X + Z_1 \beta_{Z_1} + Z_2 \beta_{Z_2} + \varepsilon \end{align}

where $X$ are some controls, and $Z_1$ and $Z_2$ are two sets of dummy ($\{0,1\}$-valued) variables. I assume that $Z_2$ does not depend upon $Z_1$, but $Z_1$ might depend upon $Z_2$ in the sense that the probability that an element of $Z_1$ might be affected by the value of $Z_2$. I would like to decompose the information (variation) in $Z_2$ into a component that affects $Y$ only through $Z_1$, and a residual that affects $Y$ directly.

If $Z_1$ were continuous, I could do something like regress (each coordinate of) $Z_1$ onto $Z_2$, then regress $Y$ on the fitted values. The natural analogue of this, say fitting a logistic (or similar) regression model for $Z_1$ gives up linearity, losing interpretability, and takes much longer to compute.

I would appreciate any insight on literature addressing this or similar problems.


In the approach you write for the case where the r.v.'s were continuous, you would regress $Z_1$ on $Z_2$ and use the fitted values $\hat Z_1$ in the $Y$-regression. But $\hat Z_1 = \hat E(Z_1\mid Z_2)$. So what you need is the conditional expectation function of $Z_1$ given $Z_2$.

This paper has a nice exposition for the bivariate Bernoulli distribution (as an introduction to the multivariate case).

Denoting the joint probability $P(Z_1 = 0, Z_2=0)=p(0,0) $ and analogously for the other three possible combinations, the conditional density of $Z_1\mid Z_2$ is (eq. 2.11 of paper) $$P(Z_1=z_1\mid Z_2=z_2) = \left(\frac {p(1,z_2)}{p(1,z_2)+p(0,z_2)}\right)^{z_1}\left(\frac {p(0,z_2)}{p(1,z_2)+p(0,z_2)}\right)^{1-z_1} $$

Then the conditional expectation function we are looking for is

$$E(Z_1\mid Z_2=z_2) = \frac {p(1,z_2)}{p(1,z_2)+p(0,z_2)} \equiv Z_1^* $$

You will then have to approximate the joint probabilities, creating a standard $2 \times 2$ contingency table of relative empirical frequencies -from which actually you can also test your dependence suspicion: theoretically, $Z_1$ and $Z_2$ will be independent if and only if $p(0,0)\cdot p(1,1) = p(0,1)\cdot p(1,0)$ and you can test this by a Fisher exact test.

Assuming that independence is rejected, then the contigency table will give you the estimated probabilities that you need and you will obtain the regressor

$$\hat Z_1^* = \frac {\hat p(1,z_2)}{\hat p(1,z_2)+\hat p(0,z_2)}$$

..which will produce a series of numbers, as for each observation $j$ of the sample we will use the ratio that corresponds to the value $z_{2j}$ of $Z_2$, i.e. if $z_{2j}=1$ we will have $$\hat z_{1j}^* = \frac {\hat p(1,1)}{\hat p(1,1)+\hat p(0,1)}$$ etc. $\hat Z_1^*$ is still a binary variable, but not taking the $\{0,1\}$ values any more.

| cite | improve this answer | |
  • $\begingroup$ I agree, this is the way to go if $Z_1$ and $Z_2$ are one-dimensional. This approach suffers some rather severe curse of dimensionality, however. Since I care only about modeling the mean of $Y$, it seems that there should be better approaches? It seems that the paper you reference gives a way of parameterizing the correlation structure. Would you mind making it clearer how I could estimate the conditional means of $Z_1$ without estimating exponentially many parameters in the dimension of $Z_2$? $\endgroup$ – user39430 Sep 19 '13 at 5:28
  • $\begingroup$ Multidimensionality needs some clarifications: you mean that $Z_1$ is a vector of r,v,'s, $\beta_{Z_1}$ is a vector of coefficients and $Z_1\beta_{Z_1}$ is their inner product? (the same question applies to $Z_2$). Also, in such a case, all elements of the random vector $Z_2$ have a dependence relationship with all elements of the random vector $Z_1$? $\endgroup$ – Alecos Papadopoulos Sep 19 '13 at 9:41
  • $\begingroup$ Yes, sorry. I said "...$Z_1$ and $Z_2$ are two sets of...", but I could have been clearer. All elements of $Z_2$ could depend upon $Z_1$ (and hopefully whatever procedure I use would give me some information on this). $\endgroup$ – user39430 Sep 19 '13 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.