# some examples of filtering the noise out of a data set

I have a data set which measures 60 data points in a second (60Hz). Clearly, I do not really need all 60 points in one second since this only generates some noise.

ABOUT MY DATA: So my data sets are primarily location/position data (x,y) coordinates that are obtained from a motion tracker. The motion tracker measurements are expected to have some noise in them (but nothing out of the ordinary). If I zoomed out of the graphs (to see the bigger picture, I am satisfied with the graphs). Zooming into time $t$ and $t+\delta$ would mean you are staring at the zigzag lines that may be attributed to errors. In the end, I would like to compute velocities, speeds, etc, without having to worry about the errors.

What are some examples of filtering techniques that would be applicable to this? I am pretty sure this would depend on the circumstances.

• Please explain what information you're trying to get from this stream of data: that is the primary consideration in selecting a procedure to filter it. – whuber Jan 17 '18 at 15:00
• I added a section on 'about my data' – cgo Jan 17 '18 at 15:17
• There is a multitude of solutions, varying greatly in computational accuracy and complexity. If the errors are truly random and independent, then a good old least squares approach might serve you best. But it also depends on the "etc." what you want to calculate. Are you interested only in the "best" result (e.g. smallest variance) of all final variables or of a particular one? The hint at Kalman filtering is particularly useful; it is a very usable version of a least squares fit.And there are tons of library implementations, so you don't even have to implement it yourself. – cherub Jan 22 '18 at 18:16

From the short description you provided I would suggest that you look into Kalman filters.

I won't bother to give you the details here since there is plenty of literature available online, e.g. the usual first go to place is Wikipedia. There are also some good books, e.g. if you work with MATLAB, I can recommend the book "Kalman Filtering, Theory and Practice Using MATLAB" by M.S. Grewal and A.P. Andrews.

It might be that a Kalman filter is a bit of an overkill for your problem, simpler alternatives would be the good old low-pass filter or a moving average (or maybe a zero lag moving average if you don't process the data in real-time).

If you use just single points for estimation, you get large variance due to measurement errors.

If you use MA (moving average) in the form $\hat X_t = \Sigma^t_{t-\tau}{\gamma_iX_i}$, you get bias due to velocity of the object, unless you suppuse that you have Brownian Motion.

If you don't mind a lag, you can use some kind of MA like $\hat X_t=\frac{1}{3}X_{t-1} + \frac{1}{3}X_{t} + \frac{1}{3}X_{t+1}$; you can include more terms or use different coefficients (corresponding to different kernel).

If you need $\hat X_t$ at the moment $t$, you have to estimate velocity from previous observations with something simple like OLS, and correct the $\hat X_t = \Sigma^t_{t-\tau}{\gamma_iX_i}$ formula for this velocity to get something like $\hat X_t = \Sigma^t_{t-\tau}{\gamma_i[X_i+ (t-i)\hat V_t]}$. The velocity estimate we used for the correction is just ad hoc. When you compute velocities for your original purposes, use $\hat X_t$.