In practical application, I have witnessed often the following practice. One observes a pair $(x_t, y_t)$ over time. Under the assumption that they are linearly related, we regress one against the other using geometric weights rather than uniform ones, i.e., the OLS minimizes $$\sum_{t=0}^\infty k^{t} (y_{T-t}- a x_{T-t}-b)^2$$ for some $k\in (0,1)$. This is very intuitive: we weight less observations far in the past. Compared to a "boxcar" weighting scheme, it has also the advantage of producing estimates that are changing smoothly over time, because observations do not fall abruptly off the observation window. However, I wonder if there is a probabilistic model underlying the relationship between $x_t$ and $y_t$ that justifies this choice.

  • $\begingroup$ Just the other day someone somewhere on one of the related StackExchange sites was commenting on this scheme as "poor man's Kalman filter". If I manage to unearth the link I will add it here. $\endgroup$ May 14, 2011 at 18:26
  • $\begingroup$ Thanks. I would like to see how this can be reframed as a Kalman filter. $\endgroup$
    – gappy
    May 14, 2011 at 23:10
  • 1
    $\begingroup$ I doubt there is a formal derivation, hence the quotes around poor man's version of adaptive parameters. $\endgroup$ May 14, 2011 at 23:20

2 Answers 2


"Linearly related" usually means

$$y_t = a x_t + b + \varepsilon_t$$

for constant $a$, $b$ and iid random errors $\varepsilon_t$, $t=0,1,\ldots,T$. One reason one would make an exponentially weighted OLS estimate is the suspicion that $a$ and $b$ might themselves be (slowly) varying with time, too. Thus we really think the correct model is

$$y_t = \alpha(t) x_t + \beta(t) + \varepsilon_t$$

for unknown functions $\alpha(t)$ and $\beta(t)$ which vary slowly (if at all) over time and we're interested in estimating their current values, $a = \alpha_T$ and $b = \beta_T$. Let's assume these functions are smooth, so we can apply Taylor's Theorem. This asserts that

$$\alpha(t) = \alpha(T) + \alpha'(t_{\alpha,t})(t-T)$$

for some $t_{\alpha,t}, 0 \le t_{\alpha,t} \lt T$, and similarly for $\beta(t)$. We think of $a$ and $b$ as being the most recent values, $\alpha_T$ and $\beta_T$, respectively. Use this to re-express the residuals:

$$y_t - (a x_t + b) = \alpha'(t_{\alpha,t})(t-T)x_t + \beta'(t_{\beta,t})(t-T) + \varepsilon_t\text{.}$$

Now a lot of hand-waving needs to occur. We will consider the entire right hand side to be random. Its variance is that of $\varepsilon_t$ plus $x_t^2(t-T)^2$ times the variance of $\alpha'(t_{\alpha,t})$ plus $(t-T)^2$ times the variance of $\beta'(t_{\beta,t})$. Those two variances are completely unknown, but (abracadabra) let's think of them as resulting from some kind of (stochastic) process in which possibly systematic (not random, but still unknown) "errors" or "variations" are accumulated from one time to the other. This would suggest an exponential change in those variances over time. Now just simplify the explicit (but essentially useless) expression for the right hand side, and absorb the quadratic terms $(t-T)^2$ into the exponential (since we're waving our hands so wildly about anyway), to obtain

$$y_t - (a x_t + b) = \delta_t$$

with the variance of $\delta_t$ equal to $\exp(\kappa(t-T))$ for some constant $\kappa$. Ignoring possible temporal correlations among the $\delta_t$ and assuming they have Normal distributions gives a log likelihood for the data proportional to

$$\sum_{t=0}^{T} k^{-t} (y_{T-t}- a x_{T-t}-b)^2$$

(plus an irrelevant constant depending only on $k$) with $k = \exp{\kappa}$. The exponentially weighted OLS procedure therefore maximizes the likelihood, assuming we know the value of $k$ (kind of like a profile likelihood procedure).

Although this entire derivation clearly is fanciful, it does show how, and approximately to what degree, the exponential weighting attempts to cope with possible changes in the linear parameters over time. It relates the parameter $k$ to the temporal rate of change of those parameters.

  • $\begingroup$ I agree on the hand-waving part... I am fine with simplifying assumptions on the time-varying form of the regression parameters, as long as they are clearly stated. Of course feel free to reference existing literature. $\endgroup$
    – gappy
    May 14, 2011 at 20:40
  • $\begingroup$ @whuber - I would say that the exponentially weighted regression is a very crude approximation for the particular model you have described. But it could well be an exact solution to a different model. For the model you describe it would be much better to include the heteroscedastic component due to variation in $\alpha(t)$ (or assume that it has no variation, and you are dealing with random intercept). You are making it seem as though geometric weighting is always sub-optimal, which is isn't. It depends on your prior information. $\endgroup$ May 16, 2011 at 8:47
  • $\begingroup$ @prob I agree, but I haven't been able to find a model that exactly justifies this approach, so I had to settle for pointing out some of the the things such a model might entail. I notice your reply doesn't make any progress in this direction, either ;-). $\endgroup$
    – whuber
    May 16, 2011 at 13:01
  • $\begingroup$ @whuber - and where do I make an approximation in my equation for it not to be exact? $\endgroup$ May 16, 2011 at 13:39
  • $\begingroup$ @probability You provide no justification: You simply announce the result I had already posted. In other words, you observe that when OLS minimizes such an expression it's really doing weighted least squares. OK, but isn't that perfectly obvious? What justifies this choice of weights? Where do they come from? $\endgroup$
    – whuber
    May 16, 2011 at 15:46

I think in that you actually mean $k^{t}$ as your weight, or that $k>1$. If $0<k<1$ and we take $k^{-t}$ as the weight then $k^{-\infty}=\infty$. So this actually weights the present observation the least. For example, if we take $k=0.5$ then $k^{0}=1,\;k^{-1}=2,\;k^{-2}=4,\dots,k^{-20}\approx 10^{6}$, and so on.

This is just stating something that you know about the how the variance changes with each observation (it gets bigger as you mover further backward in time from time $T$):

$$(y_{T-t}|x_{T-t},a,b,k,s) \sim Normal(ax_{T-t}+b,s^{2}k^{-t})$$

Denoting $Y\equiv\{y_{T},y_{T-1},\dots,y_{1}\}$ and $X\equiv\{x_{T},x_{T-1},\dots,x_{1}\}$ we have a joint log-likelihood of:

$$\log\left[p(Y|X,a,b,k,s)\right]=-\frac{1}{2}\left(T\log(2\pi s^{2} k^{-t})+\sum_{t=0}^{T-1}\frac{(y_{T-t}-ax_{T-t}-b)^{2}}{s^{2}k^{-t}}\right)$$

So in order to get the maximum likelihood estimates of $a$ and $b$ you have the following objective function:


Which is the one you seek


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.