Exponentially weighted moving linear regression

I have a problem where I need to calculate linear regression as samples come in. Is there a formula that I can use to get the exponentially weighted moving linear regression? Not sure if that's what you would call it though.

-
I say exponentially because I want to control the weight of old samples just like using weighted moving average. But I want the old samples to be less weighted, exponentially less, than newer samples –  brandon Apr 24 '11 at 18:57

migrated from math.stackexchange.comApr 24 '11 at 20:20

This question came from our site for people studying math at any level and professionals in related fields.

Sounds like what you want to do is a two-stage model. First transform your data into exponentially smoothed form using a specified smoothing factor, and then input the transformed data into your linear regression formula.

http://en.wikipedia.org/wiki/Exponential_smoothing

-
that got me where I needed thanks! –  brandon Apr 25 '11 at 15:50
Apparently this site won't let me hit the up arrow cause I'm too new, but definitely what I needed –  brandon Apr 25 '11 at 15:52
@brandon Now, you can (15 rep needed). –  chl Apr 25 '11 at 15:57
FYI, The Crane and Crotty reference (jstor.org/pss/2627674) is a valid URL, but when trying to get the PDF I get a message that "the file is damaged and could not be repaired". –  zbicyclist Apr 25 '11 at 17:34

Sure, just add a weights= argument to lm() (in case of R):

R> x <- 1:10    ## mean of this is 5.5
R> lm(x ~ 1)    ## regression on constant computes mean

Call:
lm(formula = x ~ 1)

Coefficients:
(Intercept)
5.5

R> lm(x ~ 1, weights=0.9^(seq(10,1,by=-1)))

Call:
lm(formula = x ~ 1, weights = 0.9^(seq(10, 1, by = -1)))

Coefficients:
(Intercept)
6.35

R>


Here is give 'more recent' (i.e., higher) values more weight and the mean shifts from 5.5 to 6.35. The key, if any, is the $\lambda ^ \tau$ exponential weight I compute on the fly; you can change the weight factor to any value you choose and depending on how you order your data you can also have the exponent run the other way.

You can do the same with regression models involving whichever regressors you have.

-
Never heard of R. Seems to be what I want, but I don't understand all of the syntax. Can you explain how that would work with this linear regression formula? (NΣXY - (ΣX)(ΣY)) / (NΣX^2 - (ΣX)^2). The formula works perfectly for my problem, but I'm writing a research paper and I need a more acceptable exponential moving form of this to work as samples come in. –  brandon Apr 24 '11 at 21:47
It seems like the seq command in that syntax means it keeps track of the last 10 samples. I need something like exponentially weighted moving average that decreases all past weights approaching but never reaching 0 –  brandon Apr 24 '11 at 21:49
I used 10 for the example; replace that with N where you determine N as the length of your dataset. Also, this is simply an application of weighted least squares which any decent regression textbook will cover in more detail. –  Dirk Eddelbuettel Apr 25 '11 at 1:19

If you are looking for an equation of the form

$$y=\alpha_n + \beta_n x$$

after $n$ pieces of data have come in, and you are using an exponential factor $k \ge 1$ then you could use

$$\beta_n = \frac{\left(\sum_{i=1}^n k^i\right) \left(\sum_{i=1}^n k^i X_i Y_i\right) - \left(\sum_{i=1}^n k^i X_i\right) \left(\sum_{i=1}^n k^i Y_i\right) }{ \left(\sum_{i=1}^n k^i\right) \left(\sum_{i=1}^n k^i X_i^2\right) - \left(\sum_{i=1}^n k^i X_i \right)^2}$$

and

$$\alpha_n = \frac{\left(\sum_{i=1}^n k^i Y_i\right) - \beta_n \left(\sum_{i=1}^n k^i X_i\right)}{\sum_{i=1}^n k^i} .$$

If rounding or speed become issues, this can be recast in other forms. It may also be worth knowing that for $k>1$ you have $\sum_{i=1}^n k^i = \frac{k(k^n - 1)}{k-1}$.

-