Prediction error for ARMA process Let $X(t)= \phi X_{t-12}+Z_t+\theta Z_{t-1}$ where $Z_t\sim WN(0,1)$. I need to find prediction error for projecting $X_t$ onto $H_{t-3}(X)$ (Hilbert space).
So, I know that $X_t \perp P_{H_{t-3}}X_t$ what gives me:
$\|X_t\|^2-\|P_{H_{t-3}}X_t\|^2=\langle \phi X_{t-12}+Z_t+\theta Z_{t-1}, \phi X_{t-12}+Z_t+\theta Z_{t-1}\rangle-\|P_{H_{t-3}}X_t\|^2=\sigma^2+\theta^2\sigma^2-\|P_{H_{t-3}}X_t\|^2$
but what to do with $\|P_{H_{t-3}}X_t\|^2$?
 A: Remark: Mind, that we can solve this only is $(X_t)$ is casual. A sufficient condition for that is $|\varphi|<1$. Using the representation with a lag operator $B$
\begin{equation}
(1-\varphi B^{12})X_t = (1 + \theta B)Z_t.
\end{equation}
We can evaluate the casual form of $(X_t)$, which is
\begin{equation}
X_t = (1+\theta B)\sum_{i = 0}^{\infty}\varphi^iX_{t-i}.
\end{equation}
The sum does not converge when $|\varphi| \geq 1$.
Remark: You said, you knew that $X_t\;\bot\;P_{H_{t-3}}X_t$, which is not true in general. The truth is $X_t - P_{H_{t-3}}X_t\;\bot\;P_{H_{t-3}}X_t$ and it gives as:
\begin{equation}
\text{Err}(P_{H_{t-3}}X_t) = ||X_t - P_{H_{t-3}}X_t||^2 = ||X_t||^2 - ||P_{H_{t-3}}X_t||^2.
\end{equation}
The Pythagorean theorem "applies" here with $a = X_t - P_{H_{t-3}}X_t$, $b = P_{H_{t-3}}X_t$ and $c = X_t$, in its form $a^2 = c^2 - b^2$.

As we have casuality the problem goes much more easily - have a look at $H_{t-3}$ set. It contains only those elements that can be expressed as linear combinations of $X_{t-3}, X_{t-4} \ldots X_{t-12} \ldots $. Because of causality $Z_{t}, Z_{t-1}, Z_{t-2}$ are perpendicular to $H_{t-3}$, so the forecast is just
\begin{equation}
P_{H_{t-3}}X_t = \varphi X_{t-12}.
\end{equation}
You can imagine $X_{t-12}$ as only one element that the projection operator $P_{H_{t-3}}$ do not bring to zero (as it does with perpendicular elements). It works as in usual $\mathbb{R}^3$ space with perpendicular vectors to a projective space.
Another usefull fact is that the covariance function of $(X_t)$ is:
\begin{equation}
\gamma_X(0) = \frac{\sigma^2(1 + \theta^2)}{1 - \varphi^2}.
\end{equation}
Also we have $\mathbb{E}X_t = m \in \mathbb{R}$ and:
\begin{equation}
\begin{split}
||X_t||^2 & = \langle X_t - m + m,\, X_t - m + m \rangle \\
& = \langle X_t - m,\,X_t - m \rangle + 2\langle X_t - m, \,m \rangle + \langle m,\,m\rangle \\
& = \gamma_X(0) + 2\mathbb{E}[(X_t - m)m] + m^2 \\
& = \gamma_X(0) + 2m\mathbb{E}X_t - 2m^2 + m^2 \\
& = \gamma_X(0) + m^2.
\end{split}
\end{equation}
But we can assume that $m = 0$ when we are considering projections. Simply center all variables in $(X_t)$ sequence, make a projection and shift them back by $m$, leaving an error untouched.
At this point, our problem is very simple:
\begin{equation}
\begin{split}
\text{Err}(P_{H_{t-3}}X_t) & = ||X_t||^2 - ||P_{H_{t-3}}X_t||^2 = \gamma_X(0) - ||\varphi X_{t-12}||^2  = \gamma_X(0) - |\varphi|^2 \gamma_X(0) \\
& = (1 - \varphi^2) \gamma_X(0)  = (1 - \varphi^2) \frac{\sigma^2(1 + \theta^2)}{1 - \varphi^2} = \sigma^2(1 + \theta^2).
\end{split}
\end{equation}
