Justification for using geometric weights in linear regression

Question

In practical application, I have witnessed often the following practice. One observes a pair $(x_t, y_t)$ over time. Under the assumption that they are linearly related, we regress one against the other using geometric weights rather than uniform ones, i.e., the OLS minimizes $$\sum_{t=0}^\infty k^{t} (y_{T-t}- a x_{T-t}-b)^2$$ for some $k\in (0,1)$. This is very intuitive: we weight less observations far in the past. Compared to a "boxcar" weighting scheme, it has also the advantage of producing estimates that are changing smoothly over time, because observations do not fall abruptly off the observation window. However, I wonder if there is a probabilistic model underlying the relationship between $x_t$ and $y_t$ that justifies this choice.

Answer 1

"Linearly related" usually means

$$y_t = a x_t + b + \varepsilon_t$$

for constant $a$, $b$ and iid random errors $\varepsilon_t$, $t=0,1,\ldots,T$. One reason one would make an exponentially weighted OLS estimate is the suspicion that $a$ and $b$ might themselves be (slowly) varying with time, too. Thus we really think the correct model is

$$y_t = \alpha(t) x_t + \beta(t) + \varepsilon_t$$

for unknown functions $\alpha(t)$ and $\beta(t)$ which vary slowly (if at all) over time and we're interested in estimating their current values, $a = \alpha_T$ and $b = \beta_T$. Let's assume these functions are smooth, so we can apply Taylor's Theorem. This asserts that

$$\alpha(t) = \alpha(T) + \alpha'(t_{\alpha,t})(t-T)$$

for some $t_{\alpha,t}, 0 \le t_{\alpha,t} \lt T$, and similarly for $\beta(t)$. We think of $a$ and $b$ as being the most recent values, $\alpha_T$ and $\beta_T$, respectively. Use this to re-express the residuals:

$$y_t - (a x_t + b) = \alpha'(t_{\alpha,t})(t-T)x_t + \beta'(t_{\beta,t})(t-T) + \varepsilon_t\text{.}$$

Now a lot of hand-waving needs to occur. We will consider the entire right hand side to be random. Its variance is that of $\varepsilon_t$ plus $x_t^2(t-T)^2$ times the variance of $\alpha'(t_{\alpha,t})$ plus $(t-T)^2$ times the variance of $\beta'(t_{\beta,t})$. Those two variances are completely unknown, but (abracadabra) let's think of them as resulting from some kind of (stochastic) process in which possibly systematic (not random, but still unknown) "errors" or "variations" are accumulated from one time to the other. This would suggest an exponential change in those variances over time. Now just simplify the explicit (but essentially useless) expression for the right hand side, and absorb the quadratic terms $(t-T)^2$ into the exponential (since we're waving our hands so wildly about anyway), to obtain

$$y_t - (a x_t + b) = \delta_t$$

with the variance of $\delta_t$ equal to $\exp(\kappa(t-T))$ for some constant $\kappa$. Ignoring possible temporal correlations among the $\delta_t$ and assuming they have Normal distributions gives a log likelihood for the data proportional to

$$\sum_{t=0}^{T} k^{-t} (y_{T-t}- a x_{T-t}-b)^2$$

(plus an irrelevant constant depending only on $k$) with $k = \exp{\kappa}$. The exponentially weighted OLS procedure therefore maximizes the likelihood, assuming we know the value of $k$ (kind of like a profile likelihood procedure).

Although this entire derivation clearly is fanciful, it does show how, and approximately to what degree, the exponential weighting attempts to cope with possible changes in the linear parameters over time. It relates the parameter $k$ to the temporal rate of change of those parameters.

Answer 2

I think in that you actually mean $k^{t}$ as your weight, or that $k>1$. If $0<k<1$ and we take $k^{-t}$ as the weight then $k^{-\infty}=\infty$. So this actually weights the present observation the least. For example, if we take $k=0.5$ then $k^{0}=1,\;k^{-1}=2,\;k^{-2}=4,\dots,k^{-20}\approx 10^{6}$, and so on.

This is just stating something that you know about the how the variance changes with each observation (it gets bigger as you mover further backward in time from time $T$):

$$(y_{T-t}|x_{T-t},a,b,k,s) \sim Normal(ax_{T-t}+b,s^{2}k^{-t})$$

Denoting $Y\equiv\{y_{T},y_{T-1},\dots,y_{1}\}$ and $X\equiv\{x_{T},x_{T-1},\dots,x_{1}\}$ we have a joint log-likelihood of:

$$\log\left[p(Y|X,a,b,k,s)\right]=-\frac{1}{2}\left(T\log(2\pi s^{2} k^{-t})+\sum_{t=0}^{T-1}\frac{(y_{T-t}-ax_{T-t}-b)^{2}}{s^{2}k^{-t}}\right)$$

So in order to get the maximum likelihood estimates of $a$ and $b$ you have the following objective function:

$$\sum_{t=0}^{T-1}k^{t}(y_{T-t}-ax_{T-t}-b)^{2}$$

Which is the one you seek