Is it possible to compute RMSE iteratively?

Question

I am working on continuous evaluation of a regression model on streaming data from sensors. I think that Mean Absolute Error (MAE) can be found out iteratively similar to this link for averaging. $$ MAE_{t} = \left(\frac{N_t-1}{N_t}\right) MAE_{t-1} + \left(\frac{1}{N_t}\right) \left|y_t^{true}-y_t^{pred} \right| $$ I prefer Root Mean Square Error (RMSE) over MAE, since it penalizes more for higher error values (as mentioned in Mean absolute error OR root mean squared error?)

Is there an approach to find Root Mean Square Error (RMSE) in a similar way?

I thought of following ways:

1. Keep a big window of previous values and compute RMSE continuously whenever a new data arrives. (This is expensive both in time and space complexity)

2. Keep track of squared errors i.e. $$SqSum_{t} = SqSum_{t-1}+ \left(y_t^{true}-y_t^{pred} \right)^{2}$$ and find $$ RMSE_t= \sqrt{\frac{Sum_{t}}{N_t}}$$ In this approach, I am afraid that the squared sum may overflow the number limit for larger $N_t$.

Is there a better way to find RMSE?

I would be grateful if there are approaches which give more weight to the errors in the most recent predictions.

Answer 1

An update formula in a similar vein to the one you provided for updating MAE would be:

$$ RMSE_t = \sqrt{\frac{t-1}{t}RMSE_{t-1}^2 + \frac{(y_t^{true}-y_t^{pred})^2}{t}} $$

Here I assume you have had $t$ observations through time $t$ (aka $N_t = t$ using the notation from your question). This formula never requires you to compute or store the sum of squared residuals across all your predictions.

The derivation is fairly straightforward:

\begin{align*} RMSE_t &= \sqrt{\frac{\sum_{i=1}^t (y_i^{true}-y_i^{pred})^2}{t}} \\ &= \sqrt{\frac{\sum_{i=1}^{t-1} (y_i^{true}-y_i^{pred})^2}{t} + \frac{(y_t^{true}-y_t^{pred})^2}{t}} \\ &= \sqrt{\frac{t-1}{t}\frac{\sum_{i=1}^{t-1} (y_i^{true}-y_i^{pred})^2}{t-1} + \frac{(y_t^{true}-y_t^{pred})^2}{t}} \\ &= \sqrt{\frac{t-1}{t}\bigg(\sqrt{\frac{\sum_{i=1}^{t-1} (y_i^{true}-y_i^{pred})^2}{t-1}}\bigg)^2 + \frac{(y_t^{true}-y_t^{pred})^2}{t}} \\ &= \sqrt{\frac{t-1}{t}RMSE_{t-1}^2 + \frac{(y_t^{true}-y_t^{pred})^2}{t}} \end{align*}

I would be grateful if there are approaches which give more weight to the errors in the most recent predictions.

Right now you can think of the update formula as assigning weight $\frac{t-1}{t}$ to the mean squared error of the first $t-1$ predictions and assigning weight $\frac{1}{t}$ to the mean squared error of the most recent prediction. You could simply change these two weights (while ensuring they always sum to 1) to change the degree to which you weight recent observations. One popular approach is exponential smoothing:

$$ wRMSE_t = \sqrt{(1-\alpha)wRMSE_{t-1}^2 + \alpha(y_t^{true}-y_t^{pred})^2} $$

Larger $\alpha$ values assign more weight to recent observations.