Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Suppose that $\{Y_{t}: t \in \mathbb{Z} \}$ is a stationary zero mean time series. Consider the Hilbert space $\mathcal{H}$ generated by the random variables $\{Y_t: t \in \mathbb{Z} \}$ with inner product $$ \langle X, Y \rangle = E(XY)$$ and norm $$||X||^2 = E|X|^2$$

Consider the subspace $\mathcal{M}$ generated by the random variables $\{Y_u: u \leq t \}$. Why are future values found by projecting onto the subspace $\mathcal{M}$? For example, why is $Y_{t+1}$ found by $\mathcal{P}_{\mathcal{M}}Y_{t+1}$?

share|improve this question
You must mean: "Why is the prediction of $Y_{t+1}$ (based on the information in $\mathcal M$ and relative to the natural loss function induced by the norm) obtained via the projection operator?" Correct? As currently written the question doesn't make much sense. Perhaps there are typos. – cardinal Jul 6 '12 at 16:26
Well any prediction of Y beyond t would have to involve only the the Yjs in M since that is all you know about the series at time t. – Michael Chernick Jul 6 '12 at 16:43
@Michael: The question appears to be "why (orthogonal) projection" not "why use the past", though I think an edit and some further clarification from the OP would be ideal. – cardinal Jul 6 '12 at 16:49
@cardinal from what part of H is he projecting? If he is projecting from all of H into M then wouldn't he be doing it to predict Yt+1 from the past? I think it is a question that does not deserve a why. You project because the set up is that the present is t and predicting the value for the series at t+1 can only use information from M assuming "future values" means prediction of future values. – Michael Chernick Jul 6 '12 at 16:58
@MichaelChernick: You seem to be implicitly assuming that the element in $\mathcal M$ that is chosen should be $\mathcal P_{\mathcal M} Y_{t+1}$, whereas, I am guessing this is precisely the question of the OP, i.e., what motivates the choice $P_{\mathcal M} Y_{t+1}$ versus some other element of $\mathcal M$. :) – cardinal Jul 6 '12 at 17:16

Question: "Why are future values found by projecting onto the subspace..."

Answer: Because the projection is the conditional expectation of $Y_{t+1}$ given the sigma-field generated by $Y_1,\dots,Y_t$, and it is known in time series analysis that this conditional expectation is, in a specific sense, the best predictor.

Here is a sketch of the geometry behind the first claim:

Conditional Expectation as a Projection

Take $Y$ as being $Y_{t+1}$, and $\mathscr{G}=\sigma(Y_1,\dots,Y_t)$.

share|improve this answer
(+1) Depending on exactly is intended by "Consider the subspace $\mathcal M$ generated by the random variables $\{Y_u: u \leq t\}$", $\mathcal M$ may not be (directly) associated with a $\sigma$-algebra and yet the problem makes sense. For example, $\mathcal M$ is often taken to be the set of finite linear combinations of $\{Y_u: u \leq t\}$. – cardinal Jul 9 '12 at 3:00
That's probably not what the question is asking. This is a general answer that doesn't use the Hilbert space structure at all. E.g. projection onto the the one-dimensional Hilbert subspace generated by a single r.v. $Y$ and projection onto the infinite dimensional subspace $L^2(\sigma(Y))$ (i.e. taking the conditional expectation w.r.t. $\sigma(Y)$) are clearly not the same. – Michael Dec 22 '13 at 11:21

If I assume that Cardinal's assumptions are true the reason that the predicted value for Yt+1 in the space M should be the orthogonal projection is because it is the closest point to Yt+1 in M based on the inner product metric.

share|improve this answer

"Prediction" that uses the Hilbert space structure is, as Michael Chernick says above, orthogonal projection of $Y_{t+1}$ onto the subspace generated by the "predictors" $\{ Y_u, u \leq t\}$.

This is linear regression in the population sense and not the same as orthogonal projection onto $L^2(\sigma(Y_t, Y_{t-1}, \cdots))$, i.e. conditional expectation with respect to $\sigma(Y_t, Y_{t-1}, \cdots)$. The latter subspace is in general much larger than the one generated by $\{ Y_u, u \leq t\}$.

An explicit calculation is just linear algebra. Geometrically, a Hilbert space is no different from $\mathbb{R}^n$ with the Euclidean inner product (except that it's not locally compact but that's not relevant here).

Denote the autocovariance function of ${Y_t}$ by $\gamma(h)$. One-step ahead prediction means finding $\phi_u$, $u = 1, 2, \cdots$ s.t.

$$ \|Y_{t+1} - \sum_{u \geq 1} \phi_u Y_{t+1-u}\|^2 $$

is minimized. If $\langle Y_s, Y_s + h \rangle = \gamma(h) = 0$ for all $h > p$, then

$$ \begin{bmatrix} \gamma(0) & \gamma(1) & \cdots & \gamma(p-1) \\ \gamma(1) & \gamma(0) & \vdots & \gamma(p-2) \\ \vdots & \vdots & \ddots & \vdots \\ \gamma(p-1) & \gamma(p-2) &\cdots &\gamma(0) \end{bmatrix} \begin{bmatrix} \phi_1 \\ \phi_2 \\ \vdots \\ \phi_p \end{bmatrix} = \begin{bmatrix} \gamma(1) \\ \gamma(2) \\ \vdots \\ \gamma(p) \end{bmatrix}. $$

Assuming the matrix on the left hand side is positive definite, invert it and you're done. For $n$-step ahead prediction, shift the right hand side forward by $n$.

In the general case where $\gamma$ does not have finite support but, say, is absolutely summable, taking matrices of increasing size gives an approximating sequence.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.