3
$\begingroup$

I have been reading Elements of Statistical Learning, and in Chapter 6 on Local Regression, they state the following for fitting local regression at point $x_0$ from data $\mathbf{x}$ of size $\rm{dim}(\mathbf{x}) = N$:

Define the effective kernel $l(x_0)$ (a vector of size $N$) as:

$$l(x_0)=b(x_0)^T(\mathbf{B}^T\mathbf{W}(x_0)\mathbf{B})^{-1}\mathbf{B}^T\mathbf{W}$$

Where:

$b(x_0)^T = (1,x_0)^T$

$\mathbf{B} = $ $N \times 2$ matrix with rows equal to $(1,x_i), x_i\in \mathbf{x}$

$W(x_0) = $ $N \times N$ diagonal matrix where $w_{ii} = K(x_0,x_i)$ for some kernel $K$.

They (Hastie, Tibshirani) say that it is possible to show the following:

$$\sum_1^N l_i(x_0) = 1$$

and

$$\sum_1^N (x_i-x_0)l_i(x_0) = 0$$

I tried to approach solving this problem using knowledge that $l(x_0)$ minimizes the sum of weighted squares, but with no luck -- no next step.

I also tried to see if there were some identities that indicated that $\langle l(x_0),\mathbf{1}\rangle = 1$ and $\langle l(x_0),\mathbf{x}-x_0\mathbf{1}\rangle = 0$ --- the vector of effective kernel values is orthogonal (and positive sense) to the vector of linear deviations of the associated kernel data points from our test point).

This seemed too abstract an approach and I couldn't get further.

So, it seems that we actually need to derive the formula for each element from the messy matrix equation and then derive the properties. This isn't too bad for the linear case, since we are inverting a $2\times 2$ matrix.

However, I was hoping for a more elegant approach, since the direct approach would become a beast to deal with when we try to show, for example, that a degree-$k$ local polynomial fit will only have bias from order $k+1$ and higher terms of the Taylor expansion of the underlying function being approximated.

Any help here -- is there some deep theorem in Linear Algebra that makes the issue of calculating the bias of local regression more transparent, or is it really a component-wise slog (i.e., explosion of determinants as we move upwards in degree?)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy