How to use Kalman filter in regression?

Question

I read that Kalman filter can be applied to perform regression with a dynamic beta, calculated on the fly. Can someone please break this down for me, with some simple example of single-variable regression? What are the assumptions, what are the inputs, what are the equations? By inputs I also mean user-set parameters(configs), such as half-life in the context of ema.

Answer 1

The standard Kalman filter model is given by: \begin{align*} y_t &= \mathbf{F}_t' \boldsymbol{\theta}_t + \nu_t, \qquad \nu_t \sim \mathcal{N}(0, v_t)\\ \boldsymbol{\theta}_t &= \mathbf{G}_t \boldsymbol{\theta}_{t-1} + \boldsymbol{\omega}_t, \qquad \boldsymbol{\omega}_t \sim \mathcal{N}(0, \mathbf{W}_t) \end{align*}

Say you have a pair of random variables $y_t$ and $\mathbf{F}_t$ - for example, the price of a stock and a set of covariates including the time of the year, prices of other stocks, etc. The Kalman filter assumes that the relationship between $y_t$ and $\mathbf{F}_t$ varies as a function of time. So, while today, the two might be highly correlated, tomorrow, they may not be at all (usually the dynamics are much more gradual).

To fit a Kalman filter, you use a forward filtering, backward smoothing approach. Essentially, you are assuming a prior distribution on your parameter, and based on the discrepancy between your prediction and of $y_t$ and the true value, the prior is updated.

$v_t$ controls the scale of the $y_t$, and $\mathbf{W}_t$ controls the scale of $\boldsymbol{\theta}_t$. This means that there is an inherent identifiability problem; if we don't care about the identifiability or don't know the scale of one of the variables, we can leave it. It won't affect the model fit. If we have a ballpark estimate of one or the other, we can place a prior distribution on it. If the scale is known exactly, we can fix one of the two, and leave the other to be inferred. Note that if the scale is too big, our predictions will be more or less flat. If the scale is too small, the predictions will be very jittery.

Since you're interested in doing regression "on the fly," I'll derive the forward filtering steps here. (Backward smoothing is used when you have observed all the data and want to correct parameter estimates given future data.) $v_t$, $\mathbf{W}_t$, and $\mathbf{G}_t$ must be set by the user. (The first two can be inferred but it makes the update equations a bit more complicated - see Prado and West (2010). $\mathbf{G}_t$ is usually set by knowledge of the process.) There are three distributions: prior state, prior observation (i.e. forecast), posterior state. Derivations of these are given as follows, where $\mathcal{D}_t$ is all observed data ($y_t$ and $\mathbf{F}_t)$ up to and including time $t$.

Prior state \begin{align*} \boldsymbol{\theta}_t | \mathcal{D}_{t-1} &\sim \mathcal{N}(\mathbf{a}_t, \mathbf{R}_t)\\ \mathbf{a}_t &= \mathbb{E}[\boldsymbol{\theta}_t | \mathcal{D}_{t-1}]\\ &= \mathbb{E}[\mathbf{G}_t \boldsymbol{\theta}_{t-1} + \boldsymbol{\omega}_t | \mathcal{D}_{t-1}]\\ &= \mathbf{G}_t \mathbb{E}[\boldsymbol{\theta}_{t-1} | \mathcal{D}_{t-1}]\\ & = \boxed{\mathbf{G}_t \mathbf{m}_{t-1}}\\ \mathbf{R}_t &= \mbox{Var}[\boldsymbol{\theta}_t | \mathcal{D}_{t-1}]\\ &= \mbox{Var}[\mathbf{G}_t \boldsymbol{\theta}_{t-1} + \boldsymbol{\omega}_t | \mathcal{D}_{t-1}]\\ &= \mathbf{G}_t \mbox{Var}[\boldsymbol{\theta}_{t-1} | \mathcal{D}_{t-1}] \mathbf{G}_t' + \mbox{Var}[\boldsymbol{\omega}_t | \mathcal{D}_{t-1}]\\ &= \boxed{\mathbf{G}_t \mathbf{C}_{t-1} \mathbf{G}_t' + \mathbf{W}_t} \end{align*}

Prior observation (i.e. forecast) \begin{align*} y_t | \mathcal{D}_{t-1} &\sim \mathcal{N}(f_t, Q_t)\\ f_t &= \mathbb{E}[y_t | \mathcal{D}_{t-1}]\\ &= \mathbb{E}[\mathbf{F}_t' \boldsymbol{\theta}_t + \nu_t | \mathcal{D}_{t-1}]\\ &= \boxed{\mathbf{F}_t'\mathbf{a}_t}\\ Q_t &= \mbox{Var}[y_t | \mathcal{D}_{t-1}]\\ &= \mbox{Var}[\mathbf{F}_t' \boldsymbol{\theta}_t + \nu_t | \mathcal{D}_{t-1}]\\ &= \mathbf{F}_t' \mbox{Var}[\boldsymbol{\theta}_t | \mathcal{D}_{t-1}] \mathbf{F}_t + \mbox{Var}[\nu_t | \mathcal{D}_{t-1}]\\ &= \boxed{\mathbf{F}_t' \mathbf{R}_t \mathbf{F}_t + v_t} \end{align*}

Posterior state

There are two ways to get the posterior state. The first is by conditioning on the joint distribution of $(\boldsymbol{\theta}_t, y_t | \mathcal{D}_{t-1})$. We need to get the covariance between $\boldsymbol{\theta}_t$ and $y_t$ to establish the joint distribution of $(\boldsymbol{\theta}_t, y_t | \mathcal{D}_{t-1})$. By definition, the covariance between two random quantities $\mathbf{X}$ and $\mathbf{Y}$ is $\mbox{Cov}[\mathbf{X}, \mathbf{Y}] = \mathbb{E}[\mathbf{X}\mathbf{Y}'] - \mathbb{E}[\mathbf{X}]\mathbb{E}[\mathbf{Y}]'$. Thus: \begin{align*} \mbox{Cov}[\boldsymbol{\theta}_t, y_t | \mathcal{D}_{t-1}] &= \mbox{Cov}[\boldsymbol{\theta}_t, \mathbf{F}_t' \boldsymbol{\theta}_t + \nu_t | \mathcal{D}_{t-1}]\\ &= \mathbb{E}[\boldsymbol{\theta}_t (\mathbf{F}_t' \boldsymbol{\theta}_t + \nu_t)' | \mathcal{D}_{t-1}] - \mathbb{E}[\boldsymbol{\theta}_t | \mathcal{D}_{t-1}] \mathbb{E}[\mathbf{F}_t' \boldsymbol{\theta}_t + \nu_t' | \mathcal{D}_{t-1}]\\ &= \mathbb{E}[\boldsymbol{\theta}_t \nu_t' + \boldsymbol{\theta}_t \boldsymbol{\theta}_t' \mathbf{F}_t| \mathcal{D}_{t-1}] - \mathbb{E}[\boldsymbol{\theta}_t | \mathcal{D}_{t-1}] \mathbb{E}[\boldsymbol{\theta}_t' | \mathcal{D}_{t-1}] \mathbf{F}_t \\ &= \mathbb{E}[\boldsymbol{\theta}_t \nu_t' | \mathcal{D}_{t-1}] + \mathbb{E}[\boldsymbol{\theta}_t \boldsymbol{\theta}_t' | \mathcal{D}_{t-1}]\mathbf{F}_t - \mathbb{E}[\boldsymbol{\theta}_t | \mathcal{D}_{t-1}] \mathbb{E}[\boldsymbol{\theta}_t' | \mathcal{D}_{t-1}] \mathbf{F}_t \\ &= \mathbf{0} + (\mathbb{E}[\boldsymbol{\theta}_t \boldsymbol{\theta}_t' | \mathcal{D}_{t-1}] - \mathbb{E}[\boldsymbol{\theta}_t | \mathcal{D}_{t-1}] \mathbb{E}[\boldsymbol{\theta}_t' | \mathcal{D}_{t-1}]) \mathbf{F}_t \\ &= \mbox{Var}[\boldsymbol{\theta}_t | \mathcal{D}_{t-1}]\mathbf{F}_t\\ &= \mathbf{R}_t \mathbf{F}_t\\ &= \mathbf{a}_t Q_t \end{align*}

Thus, the joint distribution of $(\boldsymbol{\theta}_t, y_t | \mathcal{D}_{t-1})$ is given by: \begin{align*} \begin{pmatrix} \boldsymbol{\theta}_t\\ y_t \end{pmatrix} \sim \mathcal{N}\left( \begin{pmatrix} \mathbf{a}_t\\ f_t \end{pmatrix} , \begin{pmatrix} \mathbf{R}_t & \mathbf{a}_t Q_t\\ \mathbf{a}_t' Q_t & Q_t \end{pmatrix} \right) \end{align*}

Then, conditioning on $y_t$, we get: \begin{align*} \boldsymbol{\theta}_t | y_t, \mathcal{D}_{t-1} &\sim \mathcal{N}(\mathbf{m}_t, \mathbf{C}_t)\\ \mathbf{m}_t &= \mathbf{a}_t + \mathbf{a}_t Q_t Q_t^{-1}(y_t - f_t)\\ &= \boxed{\mathbf{a}_t + \mathbf{a}_t e_t}\\ \mathbf{C}_t &= \mathbf{R}_t - (\mathbf{a}_t Q_t) Q_t^{-1} (Q_t \mathbf{a}_t')\\ &= \boxed{\mathbf{R}_t - \mathbf{a}_t Q_t \mathbf{a}_t'} \end{align*}

We can also find the posterior state using Bayes' theorem. Take $\mathbf{G}_t = \mathbf{I}$ for simplicity. Then, \begin{align*} p(\boldsymbol{\theta}_t | y_t, \mathbf{F}_t) &\propto p(\mathbf{F}_t, y_t | \boldsymbol{\theta}_t) p(\boldsymbol{\theta}_t)\\ &= p(y_t | \mathbf{F}_t, \boldsymbol{\theta}_t) p(\mathbf{F}_t | \boldsymbol{\theta}_t) p(\boldsymbol{\theta}_t)\\ \end{align*} Note that $\mathbf{F}_t$ is observed and deterministic while $y_t$ is observed and random. Thus, we can get rid of the $p(\mathbf{F}_t | \boldsymbol{\theta}_t)$. \begin{align*} &= \mathcal{N}(y_t | \mathbf{F}_t' \boldsymbol{\theta}_t, v_t) \mathcal{N}(\boldsymbol{\theta}_t | \mathbf{m}_{t-1}, \mathbf{R}_t)\\ \end{align*} We use $\boldsymbol{\theta}_t$ rather than $\mathbf{m}_{t-1}$ in the expression for the mean of $\mathbf{y}_t$ because we are given $\boldsymbol{\theta}_t$ in the distribution of $y_t$. We are not given $\boldsymbol{\theta}_{t-1}$ in the distribution for $\boldsymbol{\theta}_t$, however, (this is a marginal distribution) so the expected value of $\boldsymbol{\theta}_t$ is $\mathbf{m}_{t-1}$, not $\boldsymbol{\theta}_{t-1}$. \begin{align*} &= \left[(2\pi v_t)^{-1/2}\exp\left\{-\frac{1}{2v_t}(y_t - \mathbf{F}_t'\boldsymbol{\theta}_t)^2\right\}\right]\left[(2\pi)^{-p/2}|\mathbf{R}_t|^{-1/2}\exp\left\{-\frac{1}{2}(\boldsymbol{\theta}_t - \mathbf{m}_{t-1})'\mathbf{R}_t^{-1}(\boldsymbol{\theta}_t - \mathbf{m}_{t-1})\right\}\right]\\ &\propto \exp \left\{-\frac{1}{2v_t}(y_t^2 - 2y_t\boldsymbol{\theta}'\mathbf{F}_t + \boldsymbol{\theta}_t'\mathbf{F}_t\mathbf{F}_t'\boldsymbol{\theta}_t) - \frac{1}{2}(\boldsymbol{\theta}_t'\mathbf{R}_t^{-1}\boldsymbol{\theta}_t - 2\boldsymbol{\theta}_t\mathbf{R}_t^{-1}\mathbf{m}_{t-1} + \mathbf{m}_{t-1}'\mathbf{R}_t^{-1}\mathbf{m}_{t-1})\right\}\\ &\propto \exp \left\{-\frac{1}{2}\left[\frac{1}{v_t}\boldsymbol{\theta}_t'\mathbf{F}_t\mathbf{F}_t'\boldsymbol{\theta}_t - \frac{2}{v_t}y_t\boldsymbol{\theta}_t'\mathbf{F}_t + \boldsymbol{\theta}_t'\mathbf{R}_t^{-1}\boldsymbol{\theta}_t - 2\boldsymbol{\theta}_t'\mathbf{R}_t^{-1}\mathbf{m}_{t-1}\right]\right\}\\ &= \exp \left\{ -\frac{1}{2}\left[\boldsymbol{\theta}_t'\left(\frac{1}{v_t}\mathbf{F}_t\mathbf{F}_t' + \mathbf{R}_t^{-1}\right)\boldsymbol{\theta}_t - 2\boldsymbol{\theta}_t'\left(\frac{1}{v_t}y_t\mathbf{F}_t + \mathbf{R}_t^{-1}\mathbf{m}_{t-1}\right)\right]\right\}\\ &\propto \mathcal{N}(\mathbf{m}_t, \mathbf{C}_t)\\ \end{align*} where \begin{align*} \mathbf{a}_t &= \mathbb{E}[\boldsymbol{\theta}_t | \mathcal{D}_{t-1}] = \mathbf{m}_{t-1}\\ \mathbf{m}_t &= \boxed{\mathbf{C}_t\left[\frac{1}{v_t}y_t\mathbf{F}_t + \mathbf{R}_t^{-1}\mathbf{a}_{t}\right]}\\ \mathbf{C}_t &= \boxed{\left[\frac{1}{v_t}\mathbf{F}_t\mathbf{F}_t' + \mathbf{R}_t^{-1}\right]^{-1}} \end{align*}

Although the expressions for $\mathbf{m}_t$ and $\mathbf{C}_t$ look very different when derived by conditioning on the joint distribution vs. using Bayes' theorem, they are mathematically identical.

Hope this helps!

Answer 2

I believe nobody gives you a comprehensive lecture of "Kalman filter" here. So just google it and do some homework before thinking about "regression problem".

Frankly speaking, Kalman filter is consisted of two equations. System Equation (or System Model) and Observation Equation (or Observation Model). I assume you already know the difference of these two.

Kalman filter is just a filter as it called. So before you try to use it you have to formalize your problem into the mold of "Kalman filter". In this case, the problem is "regression".

So, regression.

Well, I'm not an expert of signal filtering problem. But as far as I know there are two ways to perform "regression" with Kalman filter.

Case one:

you can assume the coefficients of the regression -- so called alpha and beta -- is time varying. In this case you have to put state variables "alpha" and "beta" into your system equation. If you have some idea on how alpha and beta should be evaluated, you have to describe it as mathematical equations. It would be the system model of your Kalman Filter. Then you can connect them with regular equation of linear regression in your observation model. I assume you don't have any particular ideas on how your alpha and beta should be evaluated...so It would be a random walk with certain variance sigma.

System Equation:

beta(t) = beta(t-1) + N(0.0, sigma) --- beta equation

alpha(t) = alpha(t-1) + N(0.0, sigma) -- alpha equation

And the regression part...you have to specify, "observed" external variable x(t) in your observation equation. Usually it's should be an element of the matrix of observation equation, some form like:

y = dot(H,x)...

But I'll write it in plain (pseudo) formula.

Observation Equation:

y(t) = beta(t-1) * x(t-1) + alpha(t-1) + N(0.0, zeta)

Observable variable: y, x.

(these variable should be fed to the filter as "observed" data)

Hidden variable: alpha, beta.

(You don't have to know the level of these variables...it will be estimated by the filter.)

And run the filter! So it estimates "hidden" variable alpha and beta for you. You should hope it's not very volatile...in that case, you can not get meaningful prediction I guess (still you can get good "estimation" of alpha and beta if your hypothesis was right)

Case 2:

In this case, you know your beta and alpha is constant (or almost constant...whatever it is, it shouldn't change much beyond of your estimation/prediction horizon) but you can assume there is no way to observe variable x(t) directly. All the thing you can observe is y(t). So you need to estimate x(t) "using" regression rather than "doing" regression. So strictly speaking, this is not the solution of regression problem but maybe this is what you really want.

In this case, you have to describe how your x(t) is evaluated in your system equation. Then, use the same "Observation Equation" above but in this time x(t) don't have to be observed. But alpha and beta must be "fixed" variable. If you don't know the exact value of alpha and beta, you can optimize it with your data, I guess.

I guess there is some the other ways to do these kind of stuffs using Kalman filter. But all the things you have to do is:

1. Study basic usage of Kalman filter. You can learn how to use some package software or write your own Kalman filter in whatever language you like (I recommend python for this type of problem, by the way)

2. Formalize your problem. What is "hidden" variable should be estimated by Kalman filter? What is "observable" variable you can obtain and use for estimation of your "hidden" variable(s)? And write it as a mathematical equation...but be careful, if you want to use "Kalman filter", your equation should be linear (and the noises should be Gaussian...usually)

If you want much more complicated methods with these kind of things, google VECM or VAR models. probably, "Regression" is not the thing you really want. In that case, you have to deal with tons of mathematics...If I were you, I won't stuck my head into such complicated models!

GOOD LUCK <3<3<3