# Justification for using geometric weights in linear regression

In practical application, I have witnessed often the following practice. One observes a pair $(x_t, y_t)$ over time. Under the assumption that they are linearly related, we regress one against the other using geometric weights rather than uniform ones, i.e., the OLS minimizes $$\sum_{t=0}^\infty k^{t} (y_{T-t}- a x_{T-t}-b)^2$$ for some $k\in (0,1)$. This is very intuitive: we weight less observations far in the past. Compared to a "boxcar" weighting scheme, it has also the advantage of producing estimates that are changing smoothly over time, because observations do not fall abruptly off the observation window. However, I wonder if there is a probabilistic model underlying the relationship between $x_t$ and $y_t$ that justifies this choice.

-
Just the other day someone somewhere on one of the related StackExchange sites was commenting on this scheme as "poor man's Kalman filter". If I manage to unearth the link I will add it here. –  Dirk Eddelbuettel May 14 '11 at 18:26
Thanks. I would like to see how this can be reframed as a Kalman filter. –  gappy May 14 '11 at 23:10
I doubt there is a formal derivation, hence the quotes around poor man's version of adaptive parameters. –  Dirk Eddelbuettel May 14 '11 at 23:20

"Linearly related" usually means

$$y_t = a x_t + b + \varepsilon_t$$

for constant $a$, $b$ and iid random errors $\varepsilon_t$, $t=0,1,\ldots,T$. One reason one would make an exponentially weighted OLS estimate is the suspicion that $a$ and $b$ might themselves be (slowly) varying with time, too. Thus we really think the correct model is

$$y_t = \alpha(t) x_t + \beta(t) + \varepsilon_t$$

for unknown functions $\alpha(t)$ and $\beta(t)$ which vary slowly (if at all) over time and we're interested in estimating their current values, $a = \alpha_T$ and $b = \beta_T$. Let's assume these functions are smooth, so we can apply Taylor's Theorem. This asserts that

$$\alpha(t) = \alpha(T) + \alpha'(t_{\alpha,t})(t-T)$$

for some $t_{\alpha,t}, 0 \le t_{\alpha,t} \lt T$, and similarly for $\beta(t)$. We think of $a$ and $b$ as being the most recent values, $\alpha_T$ and $\beta_T$, respectively. Use this to re-express the residuals:

$$y_t - (a x_t + b) = \alpha'(t_{\alpha,t})(t-T)x_t + \beta'(t_{\beta,t})(t-T) + \varepsilon_t\text{.}$$

Now a lot of hand-waving needs to occur. We will consider the entire right hand side to be random. Its variance is that of $\varepsilon_t$ plus $x_t^2(t-T)^2$ times the variance of $\alpha'(t_{\alpha,t})$ plus $(t-T)^2$ times the variance of $\beta'(t_{\beta,t})$. Those two variances are completely unknown, but (abracadabra) let's think of them as resulting from some kind of (stochastic) process in which possibly systematic (not random, but still unknown) "errors" or "variations" are accumulated from one time to the other. This would suggest an exponential change in those variances over time. Now just simplify the explicit (but essentially useless) expression for the right hand side, and absorb the quadratic terms $(t-T)^2$ into the exponential (since we're waving our hands so wildly about anyway), to obtain

$$y_t - (a x_t + b) = \delta_t$$

with the variance of $\delta_t$ equal to $\exp(\kappa(t-T))$ for some constant $\kappa$. Ignoring possible temporal correlations among the $\delta_t$ and assuming they have Normal distributions gives a log likelihood for the data proportional to

$$\sum_{t=0}^{T} k^{-t} (y_{T-t}- a x_{T-t}-b)^2$$

(plus an irrelevant constant depending only on $k$) with $k = \exp{\kappa}$. The exponentially weighted OLS procedure therefore maximizes the likelihood, assuming we know the value of $k$ (kind of like a profile likelihood procedure).

Although this entire derivation clearly is fanciful, it does show how, and approximately to what degree, the exponential weighting attempts to cope with possible changes in the linear parameters over time. It relates the parameter $k$ to the temporal rate of change of those parameters.

-
I agree on the hand-waving part... I am fine with simplifying assumptions on the time-varying form of the regression parameters, as long as they are clearly stated. Of course feel free to reference existing literature. –  gappy May 14 '11 at 20:40
@whuber - I would say that the exponentially weighted regression is a very crude approximation for the particular model you have described. But it could well be an exact solution to a different model. For the model you describe it would be much better to include the heteroscedastic component due to variation in $\alpha(t)$ (or assume that it has no variation, and you are dealing with random intercept). You are making it seem as though geometric weighting is always sub-optimal, which is isn't. It depends on your prior information. –  probabilityislogic May 16 '11 at 8:47
@prob I agree, but I haven't been able to find a model that exactly justifies this approach, so I had to settle for pointing out some of the the things such a model might entail. I notice your reply doesn't make any progress in this direction, either ;-). –  whuber May 16 '11 at 13:01
@whuber - and where do I make an approximation in my equation for it not to be exact? –  probabilityislogic May 16 '11 at 13:39
@probability You provide no justification: You simply announce the result I had already posted. In other words, you observe that when OLS minimizes such an expression it's really doing weighted least squares. OK, but isn't that perfectly obvious? What justifies this choice of weights? Where do they come from? –  whuber May 16 '11 at 15:46

I think in that you actually mean $k^{t}$ as your weight, or that $k>1$. If $0<k<1$ and we take $k^{-t}$ as the weight then $k^{-\infty}=\infty$. So this actually weights the present observation the least. For example, if we take $k=0.5$ then $k^{0}=1,\;k^{-1}=2,\;k^{-2}=4,\dots,k^{-20}\approx 10^{6}$, and so on.

This is just stating something that you know about the how the variance changes with each observation (it gets bigger as you mover further backward in time from time $T$):

$$(y_{T-t}|x_{T-t},a,b,k,s) \sim Normal(ax_{T-t}+b,s^{2}k^{-t})$$

Denoting $Y\equiv\{y_{T},y_{T-1},\dots,y_{1}\}$ and $X\equiv\{x_{T},x_{T-1},\dots,x_{1}\}$ we have a joint log-likelihood of:

$$\log\left[p(Y|X,a,b,k,s)\right]=-\frac{1}{2}\left(T\log(2\pi s^{2} k^{-t})+\sum_{t=0}^{T-1}\frac{(y_{T-t}-ax_{T-t}-b)^{2}}{s^{2}k^{-t}}\right)$$

So in order to get the maximum likelihood estimates of $a$ and $b$ you have the following objective function:

$$\sum_{t=0}^{T-1}k^{t}(y_{T-t}-ax_{T-t}-b)^{2}$$

Which is the one you seek

-
Think there's a typo in your 1st eq (you've got $y_{T-t}$ on both sides of the conditional) –  JMS May 16 '11 at 5:31
@JMS - thanks for that. have edited it accordingly –  probabilityislogic May 16 '11 at 8:35