KDE for $h \rightarrow 0$ Let $K$ be a kernel and $X_1,\dots, X_n$ a sample drawn from some distribution with density $f$. The KDE of $f(x)$ is defined by $$\hat f_h(x) = \frac{1}{nh}\sum_{i=1}^nK\left(\frac{x - X_i}{h}\right).$$
I know that $\lim_{h\rightarrow 0}\mathit E[\hat f_h(x)] = f(x)$ but what is $\lim_{h\rightarrow 0}\hat f_h(x)$? I came across this question as I wondered what would happen if we assumed that $\hat f_h(x)$ satisfied some regularity conditions such that limit and expectation could be intercahnged.
 A: As $h \to 0$, for typical choices of kernel we have $\frac1h K\left( \frac{x - X_i}{h} \right) \stackrel{h \to 0}{\to} \begin{cases} \infty & x = X_i \\ 0 & x \ne X_i \end{cases}$.
So, treating the dataset $\{X_i\}$ as fixed, $\lim_{h \to 0} \hat f_h(x)$ is a function which takes value $\infty$ when $x$ exactly coincides with any of the $X_i$, and $0$ otherwise.
The quantity $\lim_{h \to 0} \mathbb E[\hat f_h(x)]$ is the expectation of the finite-bandwidth KDE, as we take a smaller and smaller bandwidth; this is a reasonable thing to work with.
Taking the KDE to zero bandwidth before you start studying its expectation, etc, though, is (a) not particularly pleasant – e.g. computing $\mathbb E[ \lim_{h \to 0} \hat f_h(x) ]$ involves taking $0 \times \infty$, etc. – and (b) isn't really a great model for what we do when we use KDE with a small but nonzero bandwidth.
It might help to think about this from the perspective not of the value of $\hat f_h$ itself, but in terms of the distribution that it represents.
$\newcommand{\PP}{\mathbb{P}}\DeclareMathOperator{\E}{\mathbb E}$
For $h>0$, KDE turns the sampled empirical distribution $\hat\PP_n$ into a continuous distribution, call it $\hat\PP_n^{(h)}$, by convolving with a kernel.
At $h=0$, that convolution is a no-op, and you just have the sampled empirical distribution.
Now, empirical distributions do a lot of things as $n \to \infty$. For instance, we do have that $\hat \PP_n \to \PP$ in the weak topology: for any bounded continuous function $g$,
$$\E_{X \sim \hat\PP_n} g(X) \to \E_{X \sim \PP} g(X).$$
Thus, simple Monte Carlo estimators work, and you can exchange various limits as you like.
But what we're asking about in KDE is the value of the density function. You can think of this as $f(x) = \E_{X \sim \PP} \delta(X - x)$. The delta function is not a bounded continuous function, and it is not true that the density of $\hat\PP_n$ converges to the density of $\PP$: the former density doesn't even exist, and exchanging limits does not work.
To talk about density functions, we need a stronger notion of convergence. Under some assumptions and if we decrease $h$ with the right rate w.r.t. $n$, KDE does provide estimators $\hat\PP_n^{(h_n)}$ which converge to $\PP$ in the right way to get density functions out. But if $h_n$ decreases too fast with $n$, and in particular if $h_n = 0$, we don't get that. This is another view on why zero-bandwidth KDE is a strange thing.
