Online Moving-Window Ridge Regression

You can use this same closed-form solution to update your $\beta$ online, even in a moving-window context. Suppose you have $m$ data points $x_1, \ldots, x_m$ and m responses $y_1, \ldots, y_m$ and your moving-window size is $L \leq m$.

Note that you can rewrite the closed-form solution as $\left(\sum_{i=1}^L x_ix_i^T + \lambda I\right)^{-1}\sum_{i=1}^L y_ix_i = A^{-1}b$.

1. Initialize $A = \sum_{i=1}^L x_ix_i^T + \lambda I$ and $b = \sum_{i=1}^L y_ix_i$.
2. A new point $x_{m+1}$ comes in.
3. Set $A = A + x_{m+1}x_{m+1}^T - x_{m-L}x_{m-L}^T$. This adds the information in the new data point to $A$ while removing the information from the old data point now outside the moving window.
4. Predict $f(x_{m+1}) = b^TA^{-1}x_{m+1}$
5. Receive $y_{m+1}$
6. Update $b = b + y_{m+1}x_{m+1} - y_{m-L}x_{m-L}$

You can also use the Sherman-Morrison formula to update $A^{-1}$ directly instead of recomputing the inverse every time. You can see this paper for more information about online moving-window ridge regression.

Change in $\beta$ Analytically

Suppose the current $\beta$ is $\beta_t = A_t^{-1}b_t$, where $A$ and $b$ are defined as above. After a new data point $x_t$ is received and an old data point $x_{t-L}$ is dropped, we have \begin{align*} \beta_{t+1} &= A_{t+1}^{-1}b_{t+1}\\ &= \left[A_t^{-1} -\frac{\left(A_t^{-1}(x_t-x_{t-L})\right)\left(A_t^{-1}(x_t-x_{t-L})\right)^T}{1+(x_t-x_{t-L})^TA_t^{-1}(x_t-x_{t-L})}\right]\left(b_t + y_tx_t - y_{t-L}x_{t-L}\right) \end{align*} This is from the Sherman-Morrison formula and the definition of $A$ and $b$ updates.

Denote $\frac{\left(A_t^{-1}(x_t-x_{t-L})\right)\left(A_t^{-1}(x_t-x_{t-L})\right)^T}{1+(x_t-x_{t-L})^TA_t^{-1}(x_t-x_{t-L})}$ by (*), then we have, \begin{align*} \beta_t - \beta_{t+1} &= A_t^{-1}b_t - A_{t+1}^{-1}b_{t+1}\\ &= A_t^{-1}b_t - \left(A_t^{-1} -(*)\right)\left(b_t + y_tx_t - y_{t-L}x_{t-L}\right)\\ &= [A_t^{-1}-(*)]y_tx_t - [A_t^{-1} - (*)]y_{t-L}x_{t-L} - (*)b_t\\ &= [A_t^{-1}-(*)](y_tx_t - y_{t-L}x_{t-L}) - (*)b_t \end{align*}

Note that if $y_tx_t = y_{t-L}x_{t-L}$ then $(*) = 0$ and $\beta_t = \beta_{t+1}$. This makes sense intuitively and is a good sanity check, the data hasn't changed so $\beta$ shouldn't.

Hope this is helpful, might add a bit more later.

Online Moving-Window Ridge Regression

You can use this same closed-form solution to update your $\beta$ online, even in a moving-window context. Suppose you have $m$ data points $x_1, \ldots, x_m$ and m responses $y_1, \ldots, y_m$ and your moving-window size is $L \leq m$.

Note that you can rewrite the closed-form solution as $\left(\sum_{i=1}^L x_ix_i^T + \lambda I\right)^{-1}\sum_{i=1}^L y_ix_i = A^{-1}b$.

1. Initialize $A = \sum_{i=1}^L x_ix_i^T + \lambda I$ and $b = \sum_{i=1}^L y_ix_i$.
2. A new point $x_{m+1}$ comes in.
3. Set $A = A + x_{m+1}x_{m+1}^T - x_{m-L}x_{m-L}^T$. This adds the information in the new data point to $A$ while removing the information from the old data point now outside the moving window.
4. Predict $f(x_{m+1}) = b^TA^{-1}x_{m+1}$
5. Receive $y_{m+1}$
6. Update $b = b + y_{m+1}x_{m+1} - y_{m-L}x_{m-L}$

You can also use the Sherman-Morrison formula to update $A^{-1}$ directly instead of recomputing the inverse every time. You can see this paper for more information about online moving-window ridge regression.

Change in $\beta$ Analytically

TBD...

You can use this same closed-form solution to update your $\beta$ online, even in a moving-window context. Suppose you have $m$ data points $x_1, \ldots, x_m$ and m responses $y_1, \ldots, y_m$ and your moving-window size is $L \leq m$.

Note that you can rewrite the closed-form solution as $\left(\sum_{i=1}^L x_ix_i^T + \lambda I\right)^{-1}\sum_{i=1}^L y_ix_i = A^{-1}b$.

1. Initialize $A = \sum_{i=1}^L x_ix_i^T + \lambda I$ and $b = \sum_{i=1}^L y_ix_i$.
2. A new point $x_{m+1}$ comes in.
3. Set $A = A + x_{m+1}x_{m+1}^T - x_{m-L}x_{m-L}^T$. This adds the information in the new data point to $A$ while removing the information from the old data point now outside the moving window.
4. Predict $f(x_{m+1}) = b^TA^{-1}x_{m+1}$
5. Receive $y_{m+1}$
6. Update $b = b + y_{m+1}x_{m+1} - y_{m-L}x_{m-L}$

You can also use the Sherman-Morrison formula to update $A^{-1}$ directly instead of recomputing the inverse every time. You can see this paper for more information about online moving-window ridge regression.

Online Moving-Window Ridge Regression

You can use this same closed-form solution to update your $\beta$ online, even in a moving-window context. Suppose you have $m$ data points $x_1, \ldots, x_m$ and m responses $y_1, \ldots, y_m$ and your moving-window size is $L \leq m$.

Note that you can rewrite the closed-form solution as $\left(\sum_{i=1}^L x_ix_i^T + \lambda I\right)^{-1}\sum_{i=1}^L y_ix_i = A^{-1}b$.

1. Initialize $A = \sum_{i=1}^L x_ix_i^T + \lambda I$ and $b = \sum_{i=1}^L y_ix_i$.
2. A new point $x_{m+1}$ comes in.
3. Set $A = A + x_{m+1}x_{m+1}^T - x_{m-L}x_{m-L}^T$. This adds the information in the new data point to $A$ while removing the information from the old data point now outside the moving window.
4. Predict $f(x_{m+1}) = b^TA^{-1}x_{m+1}$
5. Receive $y_{m+1}$
6. Update $b = b + y_{m+1}x_{m+1} - y_{m-L}x_{m-L}$

You can also use the Sherman-Morrison formula to update $A^{-1}$ directly instead of recomputing the inverse every time. You can see this paper for more information about online moving-window ridge regression.

Change in $\beta$ Analytically

Suppose the current $\beta$ is $\beta_t = A_t^{-1}b_t$, where $A$ and $b$ are defined as above. After a new data point $x_t$ is received and an old data point $x_{t-L}$ is dropped, we have \begin{align*} \beta_{t+1} &= A_{t+1}^{-1}b_{t+1}\\ &= \left[A_t^{-1} -\frac{\left(A_t^{-1}(x_t-x_{t-L})\right)\left(A_t^{-1}(x_t-x_{t-L})\right)^T}{1+(x_t-x_{t-L})^TA_t^{-1}(x_t-x_{t-L})}\right]\left(b_t + y_tx_t - y_{t-L}x_{t-L}\right) \end{align*} This is from the Sherman-Morrison formula and the definition of $A$ and $b$ updates.

Denote $\frac{\left(A_t^{-1}(x_t-x_{t-L})\right)\left(A_t^{-1}(x_t-x_{t-L})\right)^T}{1+(x_t-x_{t-L})^TA_t^{-1}(x_t-x_{t-L})}$ by (*), then we have, \begin{align*} \beta_t - \beta_{t+1} &= A_t^{-1}b_t - A_{t+1}^{-1}b_{t+1}\\ &= A_t^{-1}b_t - \left(A_t^{-1} -(*)\right)\left(b_t + y_tx_t - y_{t-L}x_{t-L}\right)\\ &= [A_t^{-1}-(*)]y_tx_t - [A_t^{-1} - (*)]y_{t-L}x_{t-L} - (*)b_t\\ &= [A_t^{-1}-(*)](y_tx_t - y_{t-L}x_{t-L}) - (*)b_t \end{align*}

Note that if $y_tx_t = y_{t-L}x_{t-L}$ then $(*) = 0$ and $\beta_t = \beta_{t+1}$. This makes sense intuitively and is a good sanity check, the data hasn't changed so $\beta$ shouldn't.

Hope this is helpful, might add a bit more later.