Simple way to solve Bayesian regression problems?

Question

My question is on Bayesian regression problems, where I want to formulate my problem as:

Find the posterior pdf of parameters $\theta$, where model $f_\theta$ relates observations $y_i = f_\theta(\theta,t_i)+\epsilon$ to the parameters $\theta$ and modeling errors $\epsilon$. Estimate $\theta$ while including prior information on $\theta$.

I emphasize that model $f_\theta(\theta,t_i)$ is not a dynamical model although time instants $t_i$ are mentioned! Function $f_\theta$ does not propagate $\theta$ through time, but is a black box computation that integrates IMU rates (accelerations and rotation rates) over time to produce a flight trajectory lasting several minutes (atmospheric entry on Mars). The IMU integration starts from initial trajectory conditions $\theta$ (position, velocity, attitude). The outputs of $f_\theta$ are several trajectory variables at times $t_i$ which I want to compare to independent observations, e.g. IMU integrated altitude after 60 seconds to be compared with a radar measurement at that time. The entire data record of the trajectory is available already. The objective is to optimize initial trajectory conditions $\theta$ to produce a good match between the IMU integrated trajectory, and independent measurements $y_i$ at times $t_i$. All measurements $y_i$ are to be used at once, meaning that in this "fitting problem" the time values are just known X-coordinates and time propagation of states or uncertainties is not required. The model $f_\theta$ just computes some values to be compared with measurements, which happen to have occurred at different times during flight.

Popular Bayesian methods are MCMC and Laplace (approximate). I am unfamiliar with these, and worried about their computational cost. However, I do know something about Kalman filtering (KF) and thought of reformulating this problem as a Kalman update step.

In KF the function $f_\theta$ would be a measurement model as it computes predicted observations (e.g. altitude at 60 s) from a prior state estimate, in this case parameter $\theta$ which represents the initial trajectory conditions. Predicted trajectory observations (such as IMU altitudes) are to be compared with actual measurements $y_i$ that are independent of the predicted observations (e.g. from radar instead of IMU). My hope is that this Kalman update step (where a gain is computed to correct $\theta$) will improve the estimate of initial trajectory state $\theta$ and provide a posterior pdf that is based on:

• measurement uncertainties $\sigma_{y_i}$ (zero-mean Gaussian)
• prior knowledge of initial trajectory conditions $\mu_{\theta_{prior}}$ and $\sigma_{\theta_{prior}}$

To me, this seems identical to a Kalman update step where the prior is a propagated state, but here is the $\theta$ vector with prior mean and covariance. An issue with my approach could be the inclusion of $\epsilon$, because as far as I know the Kalman filter does not allow to specify uncertainty on the measurement model directly, only the covariance of the prior is propagated through the (deterministic) measurement model.

Advice on other methods, such as MCMC or Laplace, are more than welcome of course. Or a MATLAB toolkit (Mathworks or 3rd party) I could use to solve this Bayesian regression problem?

Answer 1

What I am unsure about, is if there are additional limitations / assumptions of Kalman filtering that would make this approach inaccurate, or plain wrong. For example the use of a Kalman gain in the (single) correction step.

This has nothing to do with Kalman filtering as far as I can tell. A Kalman filter assumes, among other things, that you have a hidden state process, along with your time series of observations. Among other things, you should have the same amount of states as you do observations.

The only connection that I see between what you're saying here, and Kalman filtering, is that depending on the specifics of your model, you might be able to exploit Normal-Normal conjugacy. You might just be trying to do some form of Bayesian regression or a hierarchical model. Just because you are using Bayes' rule, does not mean you are using a Kalman filter.

I suggest that you start a different question that:

a.) does not mention the words "filtering" or "Kalman"

b.) you tell us more about what your specific model is. In particular, what $f_{\theta}(\theta, t_i)$ looks like.

Answer 2

From my experience, the number of states and observations does not have to be identical: observations only need to contain sufficient information to update the Kalman prediction of the state attempted to be estimated.

"Some form of Bayesian regression" is indeed what I am trying to accomplish, by casting it as a KF prediction-correction step. From the below paper on particle filtering [1], I understand the KF to be an implementation of the same Bayes' rule used in Bayesian interference:

[1] 2005 David Salmond and Neil Gordon, An introduction to particle filters (p.3-4)

http://docdro.id/rTr7xfN

The propagated state is a prior belief PDF, the measurement model is a likelihood (probability of observing a measurement given the prior state), and the actual measurement appears in the normalizing denominator. The Kalman (or Particle Filter) update equation is very similar to Bayes' rule and combines the above into a posterior PDF (in the case of time series filtering, a propagated state at time $t+\delta t$, but this does not seem essential to me). The Bayesian update equation on p.4 of [1] is can apparently be used to derive the KF update equation, if Gaussian and linear assumptions are made.

I do agree that my question is unusual, because if KF methods can indeed be used to approximate Bayesian regression problems, it might be mentioned more often. If it can be done it probably is being done, but might go under a different name or formulation.

My domain is flight dynamics and not signal processing or stochastic methods, but to my knowledge the inclusion of prior information is not possible with stochastic least-squares fitting such as most likelihood estimators. If for example the KF is a ML estimator, it would not fit my purpose. But I thought the inclusion of prior information made it a Bayesian estimator, as explained in [1].