Optimal way to estimate the irreducible error E [ V [Y | X] ]?

Question

I was reading about conditional variance in Wikipedia and then the following property showed up $$E[V[Y \mid X]]=E[(Y-f(X))^2]-E[(E[Y \mid X]-f(X))^2]$$ Which i interpret as an irreducible error, meaning that if I want to approximate the random variable $Y$ by the random variable $X$ there is a constant term that i can´t affect regardless of what f(X) i choose.
Basically my question is how to estimate that irreducible error.

I tried two different approaches:
$\quad$ First one is using different f(X) and then averaging the results, as an example: $$E[V[Y \mid X]] \simeq \sum_{i=0}^n\frac{E[(Y-X^i)^2]-E[(E[Y \mid X]-X^i)^2]}{n}$$ $\quad$Second one is to use the definition of $V[Y \mid X]$: $$E[V[Y \mid X]]=E[E[Y^2 \mid X]-E^2[Y \mid X]]=E[Y^2]-E[E^2[Y \mid X]]$$ Both approaches have the same flaw, they rely on $E[Y \mid X]$, and i don´t know how to estimate it without making assumptions on the behaviour of $Y$.

I would consider optimal, a way to calculate that parameter without relying on $E[Y \mid X]$ explicitly

Edit: I don't consider the answer i posted is the "optimal way" of estimating it since it requires a lot of computational power and a huge data set. I have the feeling that the first approach could lead to an optimal answer, i just don't know how to develop it.

Answer 1

$E(Var(Y\mid X))=E\left((Y-E(Y\mid X))^2\right)$ so you want to find a function $f(x)$ that converges to $E(Y\mid X=x)$ for every $x$ as your sample size goes to infinity. Loosely speaking, with such an $f$, $E\left((E(Y\mid X)-f(X))^2\right)$ would vanish, and the sample analog of $E\left((Y-f(X))^2\right)$ would get you an asymptotically unbiased estimate of the irreducible error.

If the true conditional mean is linear in $X$, a linear regression will do to get a suitable $f$. If it is not linear, a nonparametric method such as a random forest could work.

Answer 2

We will need the following: The response variable $Y$, the vector of explanatory variables $X$ ,and the variable $Z$ which is the residual with respect to the least square regression.

$$Z=Y-b^tX$$

We can say that $E[Y \mid X]= b^tX + E[Z \mid X]$

According to the property stated in the question, if we choose $f(X)=b^tX+E[Z \mid X]$, we get $E[V[Y \mid X]]=E[V[Z \mid X]]$

I will use $K_{AB}$ as a notation for cross covariance matrix (defined here).Note that $K_{XX}$ is a square matrix while $K_{XY}$ and $K_{XZ}$ are vectors.

To finish this setup i will remind that $b^t$ is the vector of parameters coming from linear regression $$b^t=K_{XY}^tK_{XX}^{-1}$$

Now i will continue with the second approach showed in the question

First step is to find $E[E^2[Y \mid X]]$

$$E^2[Y \mid X]=(b^tX+E[Z \mid X])^2=b^tXX^tb+2b^tXE[Z\mid X]+E^2[Z\mid X]$$

$$E^2[Y \mid X]=b^tXX^tb+2b^tXE[Z\mid X]+V[Z \mid X]-E[Z^2 \mid X]$$

$$E[E^2[Y \mid X]]=b^tE[XX^t]b+2b^tE[XZ]+E[V[Z\mid X]]-E[Z^2]$$

Second step is to revisit the definition of $E[V[Y \mid X]]$

$$E[V[Y \mid X]]=E[Y^2]-E[E^2[Y \mid X]]$$ $$E[V[Y \mid X]]=E[Y^2]+E[Z^2]-b^tE[XX^t]b-2b^tE[XZ]-E[V[Z\mid X]]$$ $$2E[V[Y \mid X]]=E[Y^2]+E[Z^2]-b^tE[XX^t]b-2b^tE[XZ]$$ $$E[V[Y \mid X]]=\frac{E[Y^2]+E[Z^2]-b^tE[XX^t]b-2b^tE[XZ]}{2}$$

An aditional step could be making it easier to compute

We will need to assume $E[X]=0$ (which is not crazy since we can detrend the data in a lot of situations) .This would imply that $E[XX^t]=K_{XX}$ and $E[XZ]=K_{XZ}$

Now we substitute b with the only value it can possibly have

$$E[V[Y \mid X]]=\frac{E[Y^2]+E[Z^2]-K_{XY}^tK_{XX}^{-1}K_{XY}-2K_{XY}^tK_{XX}^{-1}K_{XZ}}{2}$$ $$E[V[Y \mid X]]=\frac{E[Y^2]+E[Z^2]-K_{XY}^tK_{XX}^{-1}(K_{XY}+2K_{XZ})}{2}$$

If we have the sufficient amount of data we can use pointwise estimators to get the estimated value of the irreducible error.A way to double check the value in linear regression is comparing it to $R^2$ since $R^2\leq1-\frac{\text{irreducible error}}{V[Y]}$