# How is Gram-Schmidt procedure used in the following time series context?

I was reading the innovation algorithm in Brickel's Time Series Theory and Methods (page 171-172).

Let $H$ denotes a Hilbert space, $P$ denotes the projection operator and $\bar{sp}$ denotes closed span.

It mentioned that

The innovation algorithm depends on the decomposition of $H_{n}$ into $n$ orthogonal subspaces by means of the Gram-Schmidt procedure.

Then it says

As before, we define $H_n = \bar{sp}\{X_1, ..., X_n\}$ and and the one-step predictors $$\hat{X}_{n+1} = 0 \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \text{if } n = 0$$ $$\hat{X}_{n+1} = P_{H_n} X_{n+1}\;\;\;\;\;\;\; \text{if } n \geq 1$$

and we also define $\hat{X_1}:= 0$, $$H_{n} = \bar{sp}\{X_1 - \hat{X}_1, X_2 - \hat{X}_2,..., X_n - \hat{X}_n\} \;\;\;\;\;\;\;\;\;\; n \geq 1 \;\;\;\;\;\;\;\;\;\;(1)$$ so that $$\hat{X}_{n+1} = \sum^n_{j=1}\theta_{nj}(X_{n+1-j} - \hat{X}_{n+1-j})\;\;\;\;\;\;\;\;\;\;\;(2)$$

The recursive scheme for computing ${\theta_{nj}, j = 1,...,n = 1,2,...}$, is derived later on in the book.

My first question is: how does $(1)$ come about from the Gram-Schmidt procedure ? Because from my understanding, according to the Gram-Schmidt procedure, the orthogonal vectors of $H_n$ that is also a closed span should be the following: $$u_1 = X_1 = X_1 - \hat{X}_1$$ $$u_2 = X_2 - P_{H1}X_2 = X_2 - \hat{X}_2$$ $$u_3 = X_3 - P_{H1}X_3 - P_{H_2}X_3= X_3 - \hat{X}_{1+2} - \hat{X}_{2+1}$$ $$....$$ and I thought it should be $H_n = \bar{sp}\{u_1, u_2, u_3,...\}$ which is clearly different from $(1)$ above.

My second question is how does $(1)$ lead to $(2)$ , suppose that we can ignore the what exactly $\theta_{nj}$ is for now?

• Brickel may mean Brockwell and Davis. Either way, full citations of books and papers as expected in research texts and papers is always helpful. – Nick Cox Mar 27 '15 at 11:43
• Cross-posted here math.stackexchange.com/questions/1183974/… and indeed elsewhere. Even if no-one answers a question elsewhere, leaving it open carries an obligation to cross-refer to answers received, as here. – Nick Cox Mar 27 '15 at 11:45

First question

$$u_3 = X_3 - P_{H1}X_3 - P_{H_2}X_3= X_3 - \hat{X}_{1+2} - \hat{X}_{2+1}$$

is not right. Notice $H_1 \subset H_2$. $P_{H1}$ is not orthogonal to $P_{H_2}$.

Second question

It's immediate, exactly the same as, say, when the Hilbert space is $\mathbb{R}^3$. It's by induction: take $X_0$, and $X_1$, then in your notation,

$$\hat{X}_1 = \frac{\langle X_1, X_0\rangle}{\langle X_0, X_0\rangle} (X_0 - 0) = \theta (X_0 - 0) .$$

For real random variables with finite seconds moments, the inner product is expectation of the product. If there's an $X_2$, then

$$\hat{X}_2 = P_{\{ X_1, X_0 \} } X_2 = P_{\{ X_1 - \hat{X}_1, X_0 \} } X_2 = P_{ X_1 - \hat{X}_1} X_2 + P_{X_0 } X_2.$$

This is exactly the GS procedure. You see how the $\theta$'s can be computed recursively.

• can you elaborate a bit more about the answer for second question. Because I don't quite follow. Thanks. – mynameisJEFF Mar 27 '15 at 10:08
• Are you sure you're not missing some normalization in your question there? – Michael Mar 27 '15 at 10:44
• It's actually exactly GS. As you've written, though, all $\theta$'s are 1's. – Michael Mar 27 '15 at 11:08
• I understand what you are saying . But the $\theta$ s are not actually ones. Perhaps , I should attach a picture of the original piece of text later on so everybody who is interested in this problem can gain a good understanding of the question. – mynameisJEFF Mar 27 '15 at 11:16
• Never mind, I misread. The $\theta$'s are the coefficients of the projection onto $\{ X_1, \cdots, X_{n-1}\}$. Will edit answer accordingly. – Michael Mar 27 '15 at 11:25