# How to show Neyman orthogonality of the score

In section 4.1 of this paper, the authors talk about the PLR model:

\begin{aligned} &Y=D \theta_{0}+g_{0}(X)+U, \quad E_{P}[U \mid X, D]=0 \\ &D=m_{0}(X)+V, \quad E_{P}[V \mid X]=0 \end{aligned}

where the parameter of interest is the regression coefficient $$\theta_0$$. In equation (4.4), the authors provide the score function

$$\psi(W ; \theta, \eta):=\{Y-\ell(X)-\theta(D-m(X))\}(D-m(X)), \quad \eta=(\ell, m)$$ where $$W = (Y, D, X)$$ and stated that "it's easy to see that $$\theta_0$$ satisfies the orthogonality condition $$\partial_{\eta} E_{P} \psi\left(W ; \theta_{0}, \eta_{0}\right)[\eta-$$ $$\left.\eta_{0}\right]=0$$, for $$\eta_{0}=\left(\ell_{0}, m_{0}\right)$$, where $$\ell_{0}(X)=E_{P}[Y \mid X]$$."

I have two questions:

1. How did the authors arrive at the score function $$\psi(W;\theta,\eta)$$?
2. How can one show that the orthogonality condition is satisfied?

The first question is somewhat difficult to answer, because it requires speculation about the author's psychology, but let me at least try to give some intuition that it is a sensible score. Recall that what it means for $$\psi(W;\theta,\eta)$$ to be a score is that at the true values of $$\eta_0 = (\ell_0,m_0)$$, we have the moment condition

$$E[\psi(W;\theta,\eta_0)] = 0$$

Let us now write out what that entails given the $$\psi$$ defined above. We end up with the equation

$$E[(Y - \ell_0(X) - \theta_0(D-m_0(X)))\cdot(D-m_0(X))] = 0$$ Using the linearity of expectations and rearranging, we can get a closed form expression for $$\theta_0$$ given the above equation: $$\theta_0 = \frac{E[(Y-\ell_0(X))(D-m_0(X))]}{E[(D-m_0(X))^2]} = \frac{E[\mathrm{Cov}(Y,D|X)]}{E[\mathrm{Var}(D|X)]}$$

I like to think about this expression for $$\theta_0$$ in terms of the Frisch-Waugh-Lovell (FWL) theorem. Recall that this theorem states that in the linear model $$Y = \beta D + \gamma X + \varepsilon$$, $$\beta$$ is numerically equivalent to the outcome of the model $$r_Y = \beta r_D + \delta$$ where $$r_Y = Y - \mathrm{L}(Y|X_2)$$ and $$r_D = Y - \mathrm{L}(D|X_2)$$ are respectively the residuals from predicting $$Y$$ and $$D$$ using the $$X$$'s (here $$\mathrm{L}(A|B)$$ is defined to be the best linear predictor of $$A$$ given $$B$$). Recall additionally that when $$D$$ is scalar, the OLS is just the covariance of the outcome and the predictor divided by the variance of the predictor, i.e. $$\beta = \frac{\mathrm{Cov}(r_Y,r_D)}{\mathrm{Var}(r_D)}$$. Note that the expression for $$\theta$$ can thus be thought of as the "nonparametric" analogue of the FWL theorem: rather than taking the residual from the best linear predictor, we take the residual from the best predictor, i.e. the conditional expectation function.

Now, addressing your second point, let $$\delta_\ell(X),\delta_m(X)$$ be two test functions respectively perturbing $$\ell$$ and $$m$$. Then Neyman orthogonality states that for choices of $$\delta_\ell$$ and $$\delta_m$$, we have

$$\frac{\mathrm d E[\psi(W;\theta,\eta_0 + r(\delta_\ell,\delta_m))]}{\mathrm d r} = 0$$ where the derivative is taken around the point where $$r=0$$. To prove this, we can simply expand out the definition of $$\psi$$ to obtain

\begin{aligned}E[\psi(W;\theta,\eta_0 + r(\delta_\ell,\delta_m))] &= E[(Y - \ell_0(X)-r \delta_\ell(X))(D - m_0(X) - r\delta_m(X))]\\ &- \theta E[(D-m_0(X)-r\delta_m(X))^2] \end{aligned} Let us first check that the derivative of the first term is mean 0. To do so, we note that by differentiating under the expectation sign around $$r=0$$, we have \begin{aligned} \frac{\mathrm d}{\mathrm dr} E[(Y - \ell_0(X)-r \delta_\ell(X))(D - m_0(X) - r\delta_m(X))] &= E\left[\frac{\mathrm d}{\mathrm dr} (Y - \ell_0(X)-r \delta_\ell(X))(D - m_0(X) - r\delta_m(X))\right]\\ &= -E[(Y-\ell_0(X))\delta_m(X) + \delta_\ell(X)(D- m_0(X))]\\ &= -E\left[\underbrace{E[Y-\ell_0(X)|X]}_{=0} \delta_m(X)\right] - E\left[E[\delta_\ell(X)\underbrace{E[D-m_0(X)|X]}_{=0}\right] = 0 \end{aligned} Note that the two terms above are mean 0 as a result of the fact that $$\ell_0$$ and $$m_0$$ are by definition conditional expectation functions.

In light of the above, all that remains to be checked is that the limit of the third term goes to 0 as $$r\to 0$$. Specifically, we must show $$\frac{\theta\mathrm dE[(D-m_0(X)-r\delta_m(X))^2]}{\mathrm dr} = 0$$ Now, differentiating under the integral sign again, we have \begin{aligned}\frac{\mathrm d \theta E[(D-m_0(X)-r\delta_m(X))^2]}{\mathrm dr} &= \theta E\left[\frac{\mathrm d}{\mathrm dr}D-m_0(X)-r\delta_m(X))^2\right] \\&= -2 \theta E[(D-m_0(X))\delta_m(X)]\\ &= \theta E[E[(D-m_0(X))\delta_m(X) | X]]\\ &= \theta E[\underbrace{E[(D-m_0(X))|X]}_{=0}\delta_m(X)]\\ &= \theta E[0|X] = 0\end{aligned} So once again, after some manipulation, this third term equalling 0 is due to the definition of $$m_0$$ as the conditional expectation function of $$D$$.

• Thanks. For the second question, do you mean $\delta_m$ instead of $\delta_D$? Also, what exactly is $r(\delta_{\ell}, \delta_m)$ equal to? Mar 14, 2022 at 3:34
• Is $E[\psi(W, \theta_0, \eta_0 +r(\delta_{\ell}, \delta_m))]$ equal to $E[(Y- \ell_0(X) - r(\ell(X) - \ell_0(X)) - \theta_0(D-m_0(X)-r(m(X)-m_0(X)))(D - m_0(X) - r(m(X) - m_0(X))]=E[(Y- \ell_0(X) - r\delta_{\ell}(X) - \theta_0(D-m_0(X)-r\delta_m(X))(D - m_0(X) - r\delta_m(X))]?$ Mar 14, 2022 at 3:44
• Yeah exactly, the $\delta$’s are meant to represent arbitrary values of the deviations $\ell-\ell_0$ or $m-m_0$. And yeah, that’s my bad for switching notation in the middle of the derivation. Now edited. Mar 14, 2022 at 13:34
• Thanks so much. Could you also elaborate on the expansion you wrote for "To prove this, we can simply expand out the definition of $\psi$ to obtain.."? I tried expanding it myself, but I can't seem to arrive at the same result with the 3 terms. Mar 14, 2022 at 14:46
• Oh I seem to have made a mistake in my calculations. Let me rewrite that part Mar 14, 2022 at 15:13