# Consistent estimator for conditional expectation

Take sequence of random vectors $$(Y_i, X_i)_{i=1}^N$$ i.i.d.

$$X_i$$ has finite support. Let $$x$$ be a point in the support of $$X_i$$.

Consider $$E(Y_i|X_i=x)$$. Suppose it exists and is finite.

Is it true that $$\frac{\sum_{i=1}^{N} \Big[Y_i\times 1\{X_i=x\}\Big]}{\sum_{i=1}^N 1\{X_i=x\}}\rightarrow _{p} E(Y_i|X_i=x),$$ as $$N\rightarrow \infty$$? "$${1}\{...\}$$" is the indicator function taking value $$1$$ if the condition inside is satisfied and zero otherwise. "$$\to_{p}$$" denotes convergence in probability.

Does convergence also hold almost surely?

My thoughts: Regarding consistency, I've read about the Nadaraya-Watson estimator but I'm not sure that the indicator function is a valid kernel function. Regarding almost sure convergence I have no idea.

• Under the general conditions asserted, your ratio of sums may be undefined. Consider the case where the $X_i$ all have a Normal distribution, for instance: almost surely the denominator is always zero!
– whuber
Commented Jun 13, 2019 at 16:38
• I have added the condition that $X_i$ has finite support and $x$ is a point in the support. Is that enough to avoid the problem you mentioned?
– Star
Commented Jun 13, 2019 at 16:41
• Yes, it is. It seems, though, that you could simplify your question considerably by noting that the sequence of values on the left hand side changes only for those $i$ where $x$ is observed. By focusing on that subsequence and reindexing it, you are simply asking whether the sample mean of $Y_i$ converges to its expectation. May we presume you are familiar with laws of large numbers?
– whuber
Commented Jun 13, 2019 at 16:48
• Yes I am (I hope). Are you saying that, when $X_i$ is discrete, the standard (strong or weak) law of large numbers applied to a subsequence (precisely defined) of the original sequence gives convergence to the conditional expectation?
– Star
Commented Jun 13, 2019 at 16:51
• It's not quite that easy, because there is one subtlety: you need to show that such a convergent subsequence actually exists! Although it is possible for any given $x$ to be observed only finitely many times, it has to be shown that almost surely each $x$ is observed infinitely many times in any such sequence.
– whuber
Commented Jun 13, 2019 at 16:54

This is a little tricky because the number of terms in the sum is itself a random variable. I will emulate the sketch of a proof in the Wikipedia article on the (Weak) Law of Large Numbers (WLLN). This method requires only the finiteness of the conditional mean of $$Y$$ for $$X=x.$$

The basic idea was expressed in a comment to the question: the fraction is the empirical mean of the $$Y_i$$ for which $$X_i=x,$$ and therefore rightly ought to converge to the conditional expectation of $$Y.$$ However, it is possible--although increasingly unlikely--that few or even no observations of this type are observed. We cannot rule this out, but we can use its vanishing probability to carry out a proof of convergence in probability nevertheless.

The method used here exploits the fact that the number of observations where $$X_i=x$$ (among the first $$n$$ observations) has a Binomial distribution. This gives us an analytical handle on the probabilities. Rather than analyze this distribution in detail, it suffices just to apply Chebyshev's Inequality to bound the probability of low counts.

The question concerns empirical estimates of the conditional expectation of $$Y$$ for $$X=x:$$ the denominator of its fraction counts how many times $$X$$ is equal to $$x$$ in the first $$n$$ observations while the numerator sums the corresponding values of $$Y.$$ Let's call the numerator $$Q_n$$ and the denominator $$W_n.$$ Notice that $$W_n$$ has a Binomial distribution with parameters $$n$$ and $$p=\Pr(X=x).$$ Let

$$p_n(k) = \Pr(W_n=k)$$

and recall, for future use, that $$W_n$$ has expectation $$np$$ and variance $$np(1-p).$$

When $$W_n=0,$$ define $$Z_n=0$$ and otherwise let

$$Z_n = \frac{Q_n}{W_n} = \frac{\sum_{i=1}^{n} \Big[Y_i\times 1\{X_i=x\}\Big]}{\sum_{i=1}^N 1\{X_i=x\}}.$$

As a matter of notation, let $$[n] = \{1,2,\ldots, n\}$$ and for any subset $$\mathcal{I}\subset [n],$$ let $$|\mathcal {I}|$$ denote its cardinality and define

$$S_{\mathcal I}(Y) = \sum_{i\in\mathcal I} Y_i$$

to be the sum of the values of $$Y$$ with indexes in $$\mathcal I.$$ Let

$$\mathcal{I}(n,X) = \{i\in [n]\mid X_i=x\}.$$

This will enable us to deal precisely with the numerator later.

The proof of the WLLN shows that the sequence of characteristic functions of sample averages approaches the characteristic function of a constant (namely, the expectation). According to the Lévy continuity theorem, this implies convergence in distribution, which in turn implies convergence in probability because the limit is constant. Let us, then, use the Law of Iterated Expectations to compute the characteristic functions of the $$Z_n$$ from their definition:

$$\psi_n(t) = E\left[e^{itZ_n}\right] = \sum_{\mathcal{I}\subset [n]} E\left[e^{itS_{\mathcal I}(Y) / |\mathcal{I}_n|} \mid \mathcal{I}(n,X)=\mathcal{I}\right] \Pr(\mathcal{I}).$$

The independence of $$(X_i,Y_i)$$ implies $$\Pr(\mathcal{I}(n,X)=\mathcal{I})$$ depends only on $$k=|\mathcal{I}|$$ and that the characteristic function of $$S_\mathcal{I}(Y)/|\mathcal{I}|$$ also depends only on $$k$$ which, for $$k\ne0,$$ is given by

$$\psi_{S_\mathcal{I}(Y)/|\mathcal{I}|}(t) = \psi_Y\left(\frac{t}{k}\right)^k.$$

(For $$k=0$$ this expression is the characteristic function of $$0,$$ equal to the constant function $$1.$$) This permits us to collect the terms in the foregoing sum into groups according to the values of $$k;$$ each group corresponds to the event $$W_n=k.$$ Thus

$$\psi_n(t) = \sum_{k=0}^n \psi_Y\left(\frac{t}{k}\right)^k\,p_n(k).\tag{1}$$

The WLLN implies that as $$k$$ grows large, $$\psi_Y(t/k)^k$$ converges to the characteristic function of the constant $$\mu=E[Y];$$ namely, $$e^{it\mu}.$$ The problem is that no matter how large $$n$$ may become, these sums include terms with small values of $$k.$$ We handle this by showing that the small values of $$k$$ contribute vanishingly small amounts to the sums.

To this end, fix a number $$t$$ and pick $$\epsilon\gt 0$$ with the aim of finding some index $$N$$ for which $$\psi_n(t)$$ is within $$\epsilon$$ of $$e^{it\mu}$$ for all $$n \ge N:$$ this is what it takes to prove the pointwise limit of $$\psi_n(t)$$ is what we expect. Because the probabilities $$p_n(k)$$ sum to unity, we may write

$$\psi_n(t) - e^{it\mu} = \sum_{k=0}^n \left(\psi_Y\left(\frac{t}{k}\right)^k - e^{it\mu}\right)\,p_n(k).\tag{2}$$

As in the proof of the WLLN, select an $$n_0$$ for which $$\left|\psi_Y(t/k)^k - e^{it\mu}\right| \lt \epsilon/2$$

for all $$k \ge n_0.$$ Then, let $$\lambda\gt 1$$ be a number to be chosen momentarily and set $$N$$ to be any integer greater than $$\frac{1}{p}\left(\sqrt{n_0}+ \lambda \right)^2.$$ Let $$n\ge N$$ and split the sum in $$(2)$$ at $$n_0.$$ Consider the size of the second part (with larger values of $$k$$):

$$\left|\sum_{k=n_0}^n \left(\psi_Y\left(\frac{t}{k}\right)^k - e^{it\mu}\right)\,p_n(k)\right| \le \sum_{k=n_0}^n \left(\frac{\epsilon}{2}\right)\,p_n(k) \le \frac{\epsilon}{2}.$$

This leaves the problematic first part of the sum (where the values of $$k$$ are small), which we may bound with Hölder's Inequality,

\begin{aligned} \left|\sum_{k=0}^{n_0-1} \left(\psi_Y\left(\frac{t}{k}\right)^k - e^{it\mu}\right)\,p_n(k)\right| &\le \max_{0\le k \lt n_0}\left|\psi_Y\left(\frac{t}{k}\right)^k - e^{it\mu}\right|\left|\sum_{k=0}^{n_0-1} p_n(k)\right|\\ &\le 2\sum_{k=0}^{n_0-1} p_n(k). \end{aligned}\tag{3}

The final inequality follows because the magnitude of any characteristic function never exceeds $$1$$ (whence the magnitude of a difference cannot exceed $$2$$) and, of course, all the $$p_n(k)$$ are positive.

The sum on the right hand side of $$(3)$$ is just the cumulative Binomial probability

$$2\sum_{k=0}^{n_0-1} p_n(k) = 2\Pr(W_n \lt n_0).$$

Apply Chebyshev's Inequality to estimate this probability. This requires standardizing $$n_0:$$

$$\left|\frac{n_0-E[W_n]}{\sqrt{\operatorname{Var}(W_n)}}\right| = \frac{ np-n_0}{\sqrt{np(1-p)}} \gt \frac{(\sqrt{n_0}+\lambda)^2-n_0}{\sqrt{n_0}+\lambda} \gt \lambda.$$

Consequently, the Binomial probability in question cannot exceed $$1/\lambda^2.$$ Choosing $$\lambda \gt \sqrt{4/\epsilon}$$ produces

$$2\sum_{k=0}^{n_0-1} p_n(k) = 2\Pr(W_n \lt n_0) \lt \frac{2}{\lambda^2} = \frac{\epsilon}{2}.\tag{4}$$

Together, $$(2),$$ $$(3),$$ and $$(4)$$ imply

\eqalign{ \left|\psi_n(t) - e^{it\mu}\right| &\le \left|\sum_{k=0}^{n_0-1} \left(\psi_Y\left(\frac{t}{k}\right)^k - e^{it\mu}\right)\,p_n(k)\right| + \left|\sum_{k=n_0}^n \left(\psi_Y\left(\frac{t}{k}\right)^k - e^{it\mu}\right)\,p_n(k)\right| \\ &\lt \frac{\epsilon}{2} + \frac{\epsilon}{2} = \epsilon,}

demonstrating the convergence of $$\psi_n(t)$$ to $$e^{it\mu},$$ QED.

• Thank you very much. 1) Strong LLN not working? 2) Which regularity conditions on the existence of moments should I impose?
– Star
Commented Jun 16, 2019 at 20:45
• You don't need any conditions on the moments apart from the finiteness of the conditional expectation.
– whuber
Commented Jun 17, 2019 at 12:09

The above expression can be written as: $$\frac{\sum\limits_{i=1}^N I\{X_i=x\} Y_i}{N}\div \frac{\sum\limits_{i=1}^N I\{X_i=x\}}{N}$$ both $$I\{X_i=x\}Y_i$$ and $$I\{X_i=x\}$$ are i.i.d random variables with finite expectation, therefore, by the SLLN, the above expression converges a.s. to: $$\frac{E[I\{X_i=x\}Y_i]}{E[I\{X_i=x\}]}=E[Y_i|X_i=x]$$