# Hilbert spaces and time series

Suppose that $\{Y_{t}: t \in \mathbb{Z} \}$ is a stationary zero mean time series. Consider the Hilbert space $\mathcal{H}$ generated by the random variables $\{Y_t: t \in \mathbb{Z} \}$ with inner product $$\langle X, Y \rangle = E(XY)$$ and norm $$||X||^2 = E|X|^2$$

Consider the subspace $\mathcal{M}$ generated by the random variables $\{Y_u: u \leq t \}$. Why are future values found by projecting onto the subspace $\mathcal{M}$? For example, why is $Y_{t+1}$ found by $\mathcal{P}_{\mathcal{M}}Y_{t+1}$?

-
You must mean: "Why is the prediction of $Y_{t+1}$ (based on the information in $\mathcal M$ and relative to the natural loss function induced by the norm) obtained via the projection operator?" Correct? As currently written the question doesn't make much sense. Perhaps there are typos. –  cardinal Jul 6 '12 at 16:26
Well any prediction of Y beyond t would have to involve only the the Yjs in M since that is all you know about the series at time t. –  Michael Chernick Jul 6 '12 at 16:43
@Michael: The question appears to be "why (orthogonal) projection" not "why use the past", though I think an edit and some further clarification from the OP would be ideal. –  cardinal Jul 6 '12 at 16:49
@cardinal from what part of H is he projecting? If he is projecting from all of H into M then wouldn't he be doing it to predict Yt+1 from the past? I think it is a question that does not deserve a why. You project because the set up is that the present is t and predicting the value for the series at t+1 can only use information from M assuming "future values" means prediction of future values. –  Michael Chernick Jul 6 '12 at 16:58
@MichaelChernick: You seem to be implicitly assuming that the element in $\mathcal M$ that is chosen should be $\mathcal P_{\mathcal M} Y_{t+1}$, whereas, I am guessing this is precisely the question of the OP, i.e., what motivates the choice $P_{\mathcal M} Y_{t+1}$ versus some other element of $\mathcal M$. :) –  cardinal Jul 6 '12 at 17:16

Question: "Why are future values found by projecting onto the subspace..."

Answer: Because the projection is the conditional expectation of $Y_{t+1}$ given the sigma-field generated by $Y_1,\dots,Y_t$, and it is known in time series analysis that this conditional expectation is, in a specific sense, the best predictor.

Here is a sketch of the geometry behind the first claim:

Take $Y$ as being $Y_{t+1}$, and $\mathscr{G}=\sigma(Y_1,\dots,Y_t)$.

-
(+1) Depending on exactly is intended by "Consider the subspace $\mathcal M$ generated by the random variables $\{Y_u: u \leq t\}$", $\mathcal M$ may not be (directly) associated with a $\sigma$-algebra and yet the problem makes sense. For example, $\mathcal M$ is often taken to be the set of finite linear combinations of $\{Y_u: u \leq t\}$. –  cardinal Jul 9 '12 at 3:00
That's probably not what the question is asking. This is a general answer that doesn't use the Hilbert space structure at all. E.g. projection onto the the one-dimensional Hilbert subspace generated by a single r.v. $Y$ and projection onto the infinite dimensional subspace $L^2(\sigma(Y))$ (i.e. taking the conditional expectation w.r.t. $\sigma(Y)$) are clearly not the same. –  Michael Dec 22 '13 at 11:21

If I assume that Cardinal's assumptions are true the reason that the predicted value for Yt+1 in the space M should be the orthogonal projection is because it is the closest point to Yt+1 in M based on the inner product metric.

-

"Prediction" that uses the Hilbert space structure is, as Michael Chernick says above, orthogonal projection of $Y_{t+1}$ onto the subspace generated by the "predictors" $\{ Y_u, u \leq t\}$.

This is linear regression in the population sense and not the same as orthogonal projection onto $L^2(\sigma(Y_t, Y_{t-1}, \cdots))$, i.e. conditional expectation with respect to $\sigma(Y_t, Y_{t-1}, \cdots)$. The latter subspace is in general much larger than the one generated by $\{ Y_u, u \leq t\}$.

An explicit calculation is just linear algebra. Geometrically, a Hilbert space is no different from $\mathbb{R}^n$ with the Euclidean inner product (except that it's not locally compact but that's not relevant here).

Denote the autocovariance function of ${Y_t}$ by $\gamma(h)$. One-step ahead prediction means finding $\phi_u$, $u = 1, 2, \cdots$ s.t.

$$\|Y_{t+1} - \sum_{u \geq 1} \phi_u Y_{t+1-u}\|^2$$

is minimized. If $\langle Y_s, Y_s + h \rangle = \gamma(h) = 0$ for all $h > p$, then

$$\begin{bmatrix} \gamma(0) & \gamma(1) & \cdots & \gamma(p-1) \\ \gamma(1) & \gamma(0) & \vdots & \gamma(p-2) \\ \vdots & \vdots & \ddots & \vdots \\ \gamma(p-1) & \gamma(p-2) &\cdots &\gamma(0) \end{bmatrix} \begin{bmatrix} \phi_1 \\ \phi_2 \\ \vdots \\ \phi_p \end{bmatrix} = \begin{bmatrix} \gamma(1) \\ \gamma(2) \\ \vdots \\ \gamma(p) \end{bmatrix}.$$

Assuming the matrix on the left hand side is positive definite, invert it and you're done. For $n$-step ahead prediction, shift the right hand side forward by $n$.

In the general case where $\gamma$ does not have finite support but, say, is absolutely summable, taking matrices of increasing size gives an approximating sequence.

-