Assume that we have $N$ observations of i.i.d. data $(Y_i,X_i)_{i=1}^{N}$. We want to learn the model given by $Y=f(X)+\epsilon$. We use the data to estimate $\hat{f}$ using any machine learning algorithm, e.g. random forests or neural nets. We assume that our method is consistent, i.e. $\hat{f}\xrightarrow{p}f$, but we do not assume anything about the convergence rate. In particular, we do not assume that $\hat{f}$ is root-N consistent.

We consider a bias term that arises from a particular estimator (I don't want to bore you with the details). The bias term reads

$bias=\sqrt{N}\mathbb{E}\left[Y_i\left(1_{\left\{ f\left(X_{i}\right)<c\right\} }-1_{\left\{ \hat{f}\left(X_{i}\right)<c\right\} }\right)\right],$

where $c$ is a known constant.

I want to prove that $bias=o_p(1)$, i.e. as $N\xrightarrow{}\infty$, we have $bias\xrightarrow{p}0$. We can assume that $X$ has a density. If we didn't have the indicator functions, it would be impossible to show without assume root-N consistency, but I wonder whether we can show this due to the indicator function. For instance, we may assume that $\text{Pr}\left(f\left(X\right)=c\right)=0$.

Question: How would you prove $bias=o_p(1)$ (if it is at all)?


Maybe something like the following treatment:

Define the random variable

$$Z_i(N) \equiv \left(1_{\left\{ f\left(X_{i}\right)<c\right\} }-1_{\left\{ \hat{f}\left(X_{i}\right)<c\right\} }\right) \implies Z_i(N) \in \{-1,0,1\}$$

Since the estimator is consistent, we also have that

$$Z_i(N) \to_p 0 \implies Z_i(N) \to_d 0$$


$$ \text{bias} = -\mathbb{E}\left[Y_i\mid \{Z_i(N) =-1\}\right]\cdot \sqrt{N} \cdot P_N(Z_i=-1) + \mathbb{E}\left[Y_i\mid\{ Z_i(N) =1\}\right]\cdot \sqrt{N}\cdot P_N(Z_i=1)$$

So one needs for zero bias at the limit, that

$$\sqrt{N} \cdot P_N(Z_i=-1) \to 0,\qquad \sqrt{N}\cdot P_N(Z_i=1) \to 0$$


$$ \exists\, N^*, \,\delta >0 : N> N^*\implies P_N(Z_i=-1) \leq \frac{1}{\sqrt{N}\cdot N^\delta}$$

and the same for the other probability. But this requirement will eventually hold for some finite $N^*$, since these probabilities will not "jump down" to zero from some strictly positive value.

PS: Evidently, we need also to assume that the conditional expected values remain finite.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.