I was reading the innovation algorithm in Brickel's Time Series Theory and Methods (page 171-172).

Let $H$ denotes a Hilbert space, $P$ denotes the projection operator and $\bar{sp}$ denotes closed span.

It mentioned that

The innovation algorithm depends on the decomposition of $H_{n}$ into $n$ orthogonal subspaces by means of the Gram-Schmidt procedure.

Then it says

As before, we define $H_n = \bar{sp}\{X_1, ..., X_n\}$ and and the one-step predictors $$\hat{X}_{n+1} = 0 \;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\; \text{if } n = 0$$ $$\hat{X}_{n+1} = P_{H_n} X_{n+1}\;\;\;\;\;\;\; \text{if } n \geq 1$$

and we also define $\hat{X_1}:= 0$, $$H_{n} = \bar{sp}\{X_1 - \hat{X}_1, X_2 - \hat{X}_2,..., X_n - \hat{X}_n\} \;\;\;\;\;\;\;\;\;\; n \geq 1 \;\;\;\;\;\;\;\;\;\;(1)$$ so that $$\hat{X}_{n+1} = \sum^n_{j=1}\theta_{nj}(X_{n+1-j} - \hat{X}_{n+1-j})\;\;\;\;\;\;\;\;\;\;\;(2)$$

The recursive scheme for computing ${\theta_{nj}, j = 1,...,n = 1,2,...}$, is derived later on in the book.

My first question is: how does $(1)$ come about from the Gram-Schmidt procedure ? Because from my understanding, according to the Gram-Schmidt procedure, the orthogonal vectors of $H_n$ that is also a closed span should be the following: $$u_1 = X_1 = X_1 - \hat{X}_1$$ $$u_2 = X_2 - P_{H1}X_2 = X_2 - \hat{X}_2$$ $$u_3 = X_3 - P_{H1}X_3 - P_{H_2}X_3= X_3 - \hat{X}_{1+2} - \hat{X}_{2+1}$$ $$....$$ and I thought it should be $H_n = \bar{sp}\{u_1, u_2, u_3,...\}$ which is clearly different from $(1)$ above.

My second question is how does $(1)$ lead to $(2)$ , suppose that we can ignore the what exactly $\theta_{nj}$ is for now?

  • 1
    $\begingroup$ Brickel may mean Brockwell and Davis. Either way, full citations of books and papers as expected in research texts and papers is always helpful. $\endgroup$
    – Nick Cox
    Mar 27, 2015 at 11:43
  • $\begingroup$ Cross-posted here math.stackexchange.com/questions/1183974/… and indeed elsewhere. Even if no-one answers a question elsewhere, leaving it open carries an obligation to cross-refer to answers received, as here. $\endgroup$
    – Nick Cox
    Mar 27, 2015 at 11:45

1 Answer 1


First question

Your equation

$$u_3 = X_3 - P_{H1}X_3 - P_{H_2}X_3= X_3 - \hat{X}_{1+2} - \hat{X}_{2+1}$$

is not right. Notice $H_1 \subset H_2$. $P_{H1}$ is not orthogonal to $P_{H_2}$.

Second question

It's immediate, exactly the same as, say, when the Hilbert space is $\mathbb{R}^3$. It's by induction: take $X_0$, and $X_1$, then in your notation,

$$ \hat{X}_1 = \frac{\langle X_1, X_0\rangle}{\langle X_0, X_0\rangle} (X_0 - 0) = \theta (X_0 - 0) . $$

For real random variables with finite seconds moments, the inner product is expectation of the product. If there's an $X_2$, then

$$ \hat{X}_2 = P_{\{ X_1, X_0 \} } X_2 = P_{\{ X_1 - \hat{X}_1, X_0 \} } X_2 = P_{ X_1 - \hat{X}_1} X_2 + P_{X_0 } X_2. $$

This is exactly the GS procedure. You see how the $\theta$'s can be computed recursively.

  • $\begingroup$ can you elaborate a bit more about the answer for second question. Because I don't quite follow. Thanks. $\endgroup$ Mar 27, 2015 at 10:08
  • $\begingroup$ Are you sure you're not missing some normalization in your question there? $\endgroup$
    – Michael
    Mar 27, 2015 at 10:44
  • $\begingroup$ It's actually exactly GS. As you've written, though, all $\theta$'s are 1's. $\endgroup$
    – Michael
    Mar 27, 2015 at 11:08
  • $\begingroup$ I understand what you are saying . But the $\theta$ s are not actually ones. Perhaps , I should attach a picture of the original piece of text later on so everybody who is interested in this problem can gain a good understanding of the question. $\endgroup$ Mar 27, 2015 at 11:16
  • 1
    $\begingroup$ Never mind, I misread. The $\theta$'s are the coefficients of the projection onto $\{ X_1, \cdots, X_{n-1}\}$. Will edit answer accordingly. $\endgroup$
    – Michael
    Mar 27, 2015 at 11:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.