$\DeclareMathOperator*{\argmin}{arg\,min} \newcommand{\E}{\text{E}}$ This is from the Analytics Iowa LLC notes provided at http://www.public.iastate.edu/~vardeman/stat602/StatLearningNotesII.pdf, p. 16. To start, some notation and assumptions:

$$\mathbf{X} = \begin{bmatrix} \mathbf{x}^{T}_1 \\ \mathbf{x}^{T}_2 \\ \vdots \\ \mathbf{x}^{T}_N \end{bmatrix}$$ and $$\mathbf{Y} = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_N \end{bmatrix} \in \mathbb{R}^N$$ with $\mathbf{x}_i \in \mathbb{R}^p$ and $\mathbf{T} = \begin{bmatrix}\mathbf{X} & \mathbf{Y}\end{bmatrix}$. Suppose $(\mathbf{x}_i^{T}, y_i)\overset{\text{iid}}{\sim}P$ are independent of $(\mathbf{x}^{T}, y) \sim P$ for $i = 1, \dots, N$. $\E^{\mathbf{T}}$ denotes expectation with respect to $P^N$, and $\E^{\mathbf{x}}$ denotes expectation with respect to the distribution of $\mathbf{x}^{T}$.

The aim of this section is to rewrite $\E^{\mathbf{x}}\left(\E^{\mathbf{T}}\hat{f}(\mathbf{x})-\E[y\mid\mathbf{x}] \right)^2$.

Supppose that $\mathbf{T}$ is used to select a function $g_{\mathbf{T}}$ from some linear subspace, say $\mathcal{S} = \{g\}$, of the space of functions $h$ with $\E^{\mathbf{x}}(h(\mathbf{x}))^2 < \infty$, and that ultimately one uses as a predictor $$\hat{f}(\mathbf{x}) = g_{\mathbf{T}}(\mathbf{x})$$ Since linear subspaces are convex, $$g^{**}\equiv \E^{\mathbf{T}}g_{\mathbf{T}} = \E^{\mathbf{T}}f \in \mathcal{S}$$ Further, suppose that $$g^{*} \equiv \argmin_{g \in \mathcal{S}}\E^{\mathbf{x}}(g(\mathbf{x})-\E[y\mid\mathbf{x}])^2$$ the projection of (the function of $\mathbf{x}$) $\E[y \mid \mathbf x]$ onto the space $\mathcal{S}$. Then write $$h^{*}(\mathbf{x}) = \E[y \mid \mathbf{x}]-g^{*}(\mathbf{x})$$ so that $$\E[y\mid\mathbf{x}] = g^{*}(\mathbf{x}) + h^{*}(\mathbf{x})$$ enter image description here

Everything from "Since linear subspaces are convex" I am completely lost on. I have no problem understanding the definitions of the new notation above, but I fail to understand why these three expectations equal $0$. I'm familiar with the definition of a convex function, but that's it.

If there are any references that you know of for learning this background, I'd appreciate that as well.

  • 1
    $\begingroup$ The final (unfinished) statement is unclear because one cannot tell what "it's" refers to. Apart from that, the only statement with any substance is the original claim that convexity of $S$ implies the expectation of an $S$-valued random variable lies in $S$. That follows immediately from one particular definition of convexity--namely, weighted averages of elements of a convex set are in the convex set--and the definition of expectation (which is a weighted average). There's much less going on here than you might suppose! What, then, are you specifically trying to ask? $\endgroup$ – whuber Oct 17 '17 at 13:44
  • $\begingroup$ @whuber I can add more context. I'm mainly confused as to why the three expectations in the final sentence have value $0$. $\endgroup$ – Clarinetist Oct 17 '17 at 13:46
  • 1
    $\begingroup$ By construction, I believe: part of the meaning of "projection" is that it will fix all points in its image $S$, whence the restriction of $h^{*}$ to $S$ is the zero function. $\endgroup$ – whuber Oct 17 '17 at 13:52
  • $\begingroup$ @whuber I'm familiar with projection matrices in the context of linear models, in that, for example, when you multiply $\mathbf{P}_{\mathbf{X}} = \mathbf{X}(\mathbf{X}^{T}\mathbf{X})^{-}\mathbf{X}^{T}$ by $\mathbf{X}$, you get $\mathbf{X}$. I seem to also recall something about how $\hat{\mathbf{y}} = \mathbf{P}_{\mathbf{X}}\mathbf{y}$ is the closest (with respect to the $L_2$ norm) to $\mathbf{y}$ when restricted to the column space of $\mathbf{X}$. Is this the idea that is being used here? $\endgroup$ – Clarinetist Oct 17 '17 at 14:00
  • 2
    $\begingroup$ I believe the concepts are even simpler. As far as I can tell, the only property of a projection that is being used is the following. Suppose $X$ is a set and $S\subset X$. Let's say a map $h:X\to S$ is a projection when $h\mid_S$ is the identity function on $S$; that is, $h$ maps all of $X$ down to $S$ and fixes the points of $S$. When $X$ also admits a binary subtraction operation, that implies $h(s)-s=0$ for all $s\in S$. $\endgroup$ – whuber Oct 17 '17 at 14:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.