I am trying to understand how Parzen window density estimate converges to actual density function f(x).[Actually i am trying to learn machine learning on my own using available free resources. Please help me in the below]

Let $f_n(x)$ be the Parzen window density estimate of actual density f(x). Given $x_1,x_2....x_n $ are iid sample (given training data).

Let h be parameter. $V_n$ be volume (say hypercube). Now in Parzen, we take estimate for density function to be linear sum of kernel functions at sample points. To show that estimate converges to actual f(x), i did in following way(for each sample size n, $V_n, h_n$ , varies and also as $n \to \infty, h_n \to 0, V_n \to 0, $ but $n V_n \to 0$)

$E(\hat f_n(x)) = \frac {1}{n} \sum_{i=1}^n E(\frac{1}{V_n} \phi(\frac{x-x_i}{h_n})) \\ = E(\frac{1}{V_n} \phi(\frac{x-x_i}{h_n})) \\= \int \frac{1}{V_n} \phi(\frac{x-x_i}{h_n}) f(z) dz$ (as each term expectation is same and $\phi $ be some kernel function, f be density )

Above Last integral be (1)

After that how to proceed? I am following https://www.youtube.com/watch?v=esoVuEG-X1I&list=PLbMVogVj5nJSlpmy0ni_5-RgbseafOViy&index=13&t=2617s (At 26.01 )

Here sir says this integral (1) goes to f(x) as $n \to \infty$ but i did not understand how.

I know

$\int \frac{1}{V_n}\phi(\frac{x-x_i}{h_n}) dz = 1$ (since $\phi$ is kernel function)

Also, i tried to expand on final integral(1) using integration by parts

then $f(z)\int \frac{1}{V_n}\phi(\frac{x-x_i}{h_n}) dz - \int f'(z) \int \frac{1}{V_n}\phi(\frac{x-x_i}{h_n}) dz dz = f(z) - f(z)=0$ (as integral of kernel function sums to 1)

Please explain where i did it wrongly or understood wrongly.


First, they define the following function: $$\delta_n(x)=\frac{1}{V_n}\phi\left({x\over h_n}\right)$$ and this is assumed (check 22m39s) to converge to delta function as $n\rightarrow\infty$. So, basically any kernel that doesn't satisfy this convergence property, even if it converges to the true density, we can't prove it this way. That said,

$$E[\hat f_n (x)]=\int_{-\infty}^\infty \delta_n(x-z)f(z)dz\rightarrow \int_{-\infty}^\infty \delta(x-z)f(z)dz=\int_{x^-}^{x^+}\delta(x-z)f(z)dz=f(x)$$

The last two follows from the properties of delta function.

  • 1
    $\begingroup$ thank you sir. I was puzzled at last but 1 step. I understood after going thru shifting property of delta function(missed that approximation of f in that epsilon neighbourhood of z). $\endgroup$ – Nascimento de Cos Mar 18 '20 at 14:25
  • $\begingroup$ Sir any idea why i am getting 0 if i integrate by parts in the last part of my post(should not become zero rigt?) $\endgroup$ – Nascimento de Cos Mar 18 '20 at 14:50
  • $\begingroup$ Your expansion seems quite complicated, because you don't use limits, and confuse $x_i$ and $z$'s. I feel that first integral will be $0$. $\endgroup$ – gunes Mar 19 '20 at 14:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.