# How to calculate the expectation of the KDE using little-o?

This is possibly a duplicate of this question of mine, however, here I ask for clarification regarding an estimation that is done when calculating the expectation of the kernel density estimator (KDE) using little-o. The conditions on the KDE stated here are inspired by an exercise in an undergraduate textbook.

## Background

Suppose $$x_1, ..., x_n$$ are independent and identically distributed observations of a random variable $$X$$ with unknown distribution function $$F$$ and probability density function $$f\in C^m$$, for some $$m>1$$ fixed. Let $$k\in C^{m+1}$$ be a given fixed function such that \begin{align} k&\geq 0, \\ \mathrm{supp} (k)&=[-1,1], \\ \int_{\mathbb{R}} k(u)\mathrm{d}u&=1, \\ \int_{\mathbb{R}} k(u)u^l\mathrm{d}u&=0 \ \text{for all} \ 1\leq l Define the KDE $$f_n$$ of $$f$$ by $$f_n(t)=\frac{1}{n}\sum_{i=1}^n \frac{1}{h}k\left(\frac{t-x_i}{h}\right),$$ where $$h=h(n)$$ is the bandwidth. What is the expectation of $$f_n$$, i.e. $$\mathbb{E}[f_n(t)]$$?.

By linearity of the expectation, identical distribution of $$x_1,...,x_n$$, the law of the unconscious statistician and the change of variables $$u=(t-x)/h$$, \begin{align} \mathbb{E}[f_n(t)]&=\frac{1}{n}\sum_{i=1}^n \mathbb{E}\left[\frac{1}{h}k\left(\frac{t-x_i}{h}\right)\right]\\ &=\mathbb{E}\left[\frac{1}{h}k\left(\frac{t-x}{h}\right)\right]\\ &=\int_{\mathbb{R}}\frac{1}{h}k\left(\frac{t-x}{h}\right)f(x)\mathrm{d}x\\ &=\int_{\mathbb{R}}\frac{1}{h}k(u)f(t-hu)h\mathrm{d}u\\ &=\int_{\mathbb{R}}k(u)f(t-hu)\mathrm{d}u. \tag{1} \end{align} From $$f\in C^m$$, it follows that $$f(t-hu)=\sum_{l=0}^m \frac{f^{(l)}(t)}{l!} (-hu)^l+o((hu)^m). \tag{2}$$ $$o(g(y))$$ is a set (or a function) such that $$f(y)\in o(g(y))$$ (or $$f(y)=o(g(y))$$) satisfies $$\lim_{y\to y_0} f(y)/g(y)=0$$ for $$y_0$$ denoting a real number, a complex number or $$\pm \infty$$. In $$(2)$$, $$y_0=0$$. From $$(1)$$, $$(2)$$ and linearity of integration, \begin{align} \mathbb{E}[f_n(t)]&=\int_{\mathbb{R}}k(u)\left(\sum_{l=0}^m \frac{f^{(l)}(t)}{l!} (-hu)^l+o((hu)^m)\right)\mathrm{d}u \\ &=\sum_{l=0}^m\int_{\mathbb{R}}k(u)\frac{f^{(l)}(t)(-hu)^l}{l!}\mathrm{d}u+\int_{\mathbb{R}}k(u)o((hu)^m)\mathrm{d}u. \tag{3} \end{align} From the given conditions on $$k$$, the $$l=0$$ term reads $$\int_{\mathbb{R}} k(u)f(t)\mathrm{d}u=f(t)\int_{\mathbb{R}} k(u) \mathrm{d}u=f(t).$$ The $$1\leq l terms are $$\int_{\mathbb{R}} k(u)\frac{f^{(l)}(t)}{l!} (-hu)^l\mathrm{d}u=\frac{f^{(l)}(t)(-h)^l}{l!}\int_{\mathbb{R}} k(u)u^l\mathrm{d}u=0.$$ Finally, the $$l=m$$ term is $$\frac{f^{(m)}(t)(-h)^m}{m!}\int_{\mathbb{R}} k(u)u^m\mathrm{d}u<\infty.$$ From the definition of $$o(g(y))$$ given above, $$o((hu)^m)$$ denotes a function (the remainder) of $$h$$ and $$u$$ that for small $$hu$$, i.e. $$hu\to 0$$, approaches $$0$$ faster than $$(hu)^m$$. The remainder appears under the integral sign of an improper integral, which is the limit of a definite integral. For finite $$u$$, $$hu\to 0$$ means $$h\to 0$$. Thus the remainder is not only in $$o((hu)^m)$$ but also non-uniformly in $$o(h^m)$$, that is, for the remainder it holds that $$o((hu)^m)=u^mo(h^m)=o(h^m)$$.

## Question

In the answer to the above linked question it is claimed, with slight modification, that $$$$\int_\mathbb{R} k(u) o((hu)^m)\mathrm{d}u = \int_\mathbb{R} k(u) o(h^m)\mathrm{d}u =o(h^m)\int_\mathbb{R} k(u) u^m\mathrm{d}u =o(h^m) \tag{4},$$$$ but if the remainder is non-uniformly in $$o(h^m)$$, then the last two equalities in $$(4)$$ may not hold. The following example shows how a similar reasoning may fail.

For each positive $$a$$ and $$x$$ near $$0$$, $$$$g(x,a)=\frac{x^2}{a^2+x^2}\in o\!\left(x^{3/2}\right). %\ \text{for} \ x \ \text{near} \ 0.$$$$ Define $$$$f(x)=\int_0^1g(x,a)\,\mathrm{d}a.$$$$ Is $$f(x)\in o\!\left(x^{3/2}\right)$$? It is tempting to reason as in $$(4)$$; $$$$\int_0^1g(x,a)\,\mathrm{d}a=\int_0^1o\!\left(x^{3/2}\right)\mathrm{d}a=o\!\left(x^{3/2}\right).$$$$ However, $$\lim_{x\to0}f(x)/x=\pi/2$$, which means that $$f(x)\not\in o\!\left(x\right)\supseteq o\!\left(x^{3/2}\right)$$.
So, for some $$g(h,u)\in o(1)$$1, $$\int_\mathbb{R} k(u) o((hu)^m)\mathrm{d}u= h^m \int_\mathbb{R} k(u) u^m g(h,u)\mathrm{d}u,$$ but without knowing how $$g(h,u)$$ behaves away from zero, it seems like no further estimates can be done. Is it possible to calculate the expectation of the KDE using little-o?

Footnotes:

1. The notation $$g(h,u)$$ implies the notation $$g(hu)$$. Unlike $$g(hu)$$, $$g(h,u)$$ includes those functions in $$o(1)$$ where $$h$$ and $$u$$ not only appear as $$hu$$.
• Related; here.
– schn
Jan 3 at 22:53

Here is a suggested solution:

1. The integration occurs over $$[-1,1]$$ due to $$\mathrm{supp}(k)=[-1,1]$$ and $$f$$ being a probability density function, i.e. it integrates to $$1$$ over $$\mathbb{R}$$ and is thus bounded.

From the definition of $$o(g(y))$$ given above, $$o((hu)^m)$$ denotes a function (the remainder) of $$h$$ and $$u$$ that for small $$hu$$, i.e. $$hu\to 0$$, approaches $$0$$ faster than $$(hu)^m$$. The remainder appears under the integral sign of an improper integral, which is the limit of a definite integral. For finite $$u$$, $$hu\to 0$$ means $$h\to 0$$. Thus the remainder is not only in $$o((hu)^m)$$ but also non-uniformly in $$o(h^m)$$, that is, for the remainder it holds that $$o((hu)^m)=u^mo(h^m)=o(h^m)$$.
Regarding the integral with the remainder, note that $$o((hu)^m)$$ does not specify the remainder; it only specifies that the remainder converges to $$0$$ faster than $$(hu)^m$$ as $$hu\to 0$$. $$h$$ is the free variable while $$u$$ is the bounded variable; $$hu\to 0$$ means $$h\to 0$$. Thus the remainder satisfies $$o((hu)^m)=u^mo(h^m)=o(h^m)$$. If the remainder is only a function of $$h$$, the estimation of the integral is straightforward. If it also depends on $$u$$, then the integral can be estimated if the remainder, viewed as a sequence of functions, converges uniformly to $$0$$1. Then $$$$\int_{[-1,1]} k(u) o((hu)^m)\mathrm{d}u =\int_{[-1,1]} k(u) o(h^m) \mathrm{d}u=o(h^m).$$$$
1. If a sequence of functions $$g_n$$ converges uniformly to a function $$g$$ over some compact interval $$I$$ where $$g_n$$ and $$g$$ are integrable, then $$$$\label{uniform} \lim_{n\to\infty}\int_I g_n(u)\mathrm{d}u=\int_I g(u)\mathrm{d}u.$$$$