Computation of multiple linear OLS regression with rolling window and/or update How can I efficiently calculate an OLS fit for N multiple variables for a rolling window?
I've worked out how to do it for 1 and 2 variable linear fits, I'd like to extend to the general case of N variables if possible (or at least to 3). I don't want to calculate the whole regression for each individual window, I want to update my calculation with each new data point.
For example, for 1 variable fit of the form: $y_i = \alpha + \beta x_i + \epsilon_i$ (where $\epsilon$ is the error term)
$$\beta = (n\sum_{i=1}^{n} x_i y_i - \sum_{i=1}^{n} x_i \sum_{i=1}^{n} y_i) / (n \sum_{i=1}^{n} x_i^2 - (\sum_{i=1}^{n} x_i)^2)$$
$$\alpha = (\sum_{i=1}^{n} y_i - \beta \sum_{i=1}^{n} x_i)/n$$
In this case I only need to maintain rolling values of $\sum x$, $\sum y$, $\sum xy$, and $\sum x^2$ and I can easily calculate the fit parameters quickly at each window step without redoing the whole thing.
I achieved the same for a 2 variable multiple fit of the form: $y_i = \alpha + \beta_{1} x_{i1} + \beta_{2} x_{i2} + \epsilon_i$
It's pretty messy:
$$\sum X_1Y = \sum_{i=1}^{n} x_{i1}y_i - \frac{1}{n}(\sum_{i=1}^{n}x_{i1} \sum_{i=1}^{n} y_i)$$
$$\sum X_2Y = \sum_{i=1}^{n} x_{i2}y_i - \frac{1}{n}(\sum_{i=1}^{n}x_{i2} \sum_{i=1}^{n} y_i)$$
$$\sum X_1X_2 = \sum_{i=1}^{n} x_{i1}x_{i2} - \frac{1}{n}(\sum_{i=1}^{n}x_{i1} \sum_{i=1}^{n} x_{i2})$$
$$\sum X_1X_1 = \sum_{i=1}^{n} x_{i1}^2 - \frac{1}{n}(\sum_{i=1}^{n}x_{i1})^2$$
$$\sum X_2X_2 = \sum_{i=1}^{n} x_{i2}^2 - \frac{1}{n}(\sum_{i=1}^{n}x_{i2})^2$$
we then get
$$d = (\sum X_1X_1   \sum X_2X_2) - (\sum X_1X_2)^2$$
$$\beta_1 = \frac{1}{d}(\sum X_2X_2  \sum X_1Y - \sum X_1X_2  \sum X_2Y)$$
$$\beta_2 = \frac{1}{d}(\sum X_1X_1 \sum X_2Y - \sum X_1X_2  \sum X_1Y)$$
Again, here I only need to maintain rolling sums of $\sum x_1$, $\sum x_2$, $\sum y$, $\sum x_1y$, $\sum x_2y$, $\sum x_1x_2$, $\sum x_1^2$, and $\sum x_2^2$ to calculate the regression, I don't need to do the fit individually for each window, and so it's really quick. (Note I only care about the $\beta$ values).
I now want to extend this to 3 variables (and possibly more). I've started working through the algebra and it's hairy. I feel sure I must be missing a trick or a pattern here that makes the extension to 3+ variables relatively easy.
Can anyone suggest how I can extend my technique above to 3 variables (or possibly more) or otherwise to efficiently calculate the $\beta$ for a rolling window? Thanks!
 A: The formulas may be completely correct but not so easy to extend. It'd be better to consider appropriate matrix algebra. This paper (especially the first 7pages) has all the formulas necessary for an increasing-window regression via fast updates (Hostetter, Recursive Estimation).  If you download the reference you'll see it's basically an application of this 'Sherman-Morrison-Woodbury' formula for updating inverse matrices.
The regression they consider is $$z = Hx + v$$ (funny notation but it's a Signal Processing publication). The updating equations are Kalman-filter-like but simpler:
The variables are $x$ is the regression coefficients and $\hat{x}$ our estimate, $h$ is the observed independent data. $z$ is the observed (scalar) dependent data and $P = (H'H)^{-1}$ is part of the covariance of the estimator $\hat{x}$.
Measurement model (for scalar measurements)
$$z_{k+1}=  h^T (k + 1 ) x ( k + 1 ) + v_{k+1} $$
Predictor-corrector
$$\hat{x}(k + 1) = \hat{x}(k) + K(k + 1)[z_{k+1} - h^T(k + 1)\hat{x}(k)]$$
Corrector gain
$$K(k + 1) = P(k)h(k + 1)c(k + 1)$$
Gain quantities
$$c(k + 1) = [h^T(k + 1)P(k)h(k + 1) + 1 ] ^{-1}$$
$$P(k + 1) = P(k) - P(k)h(k + 1)c(k + 1)h^T(k + 1)P(k)$$
or
$$P(k+1) = [ I - K (k+ 1)h^T(k+ 1)]P(k)$$
Basically, it is more straightforward to deal with your updating equations when using matrix algebra
