13
$\begingroup$

I am proposing to try and find a trend in some very noisy long term data. The data is basically weekly measurements of something which moved about 5mm over a period of about 8 months. The data is to 1mm accuracey and is very noisy regularly changing +/-1 or 2mm in a week. We only have the data to the nearest mm.

We plan to use some basic signal processing with a fast fourier transform to separate out the noise from the raw data. The basic assumption is if we mirror our data set and add it to the end of our existing data set we can create a full wavelength of the data and therefore our data will show up in a fast fourier transform and we can hopefully then separate it out.

Given that this sounds a little dubious to me, is this a method worth purusing or is the method of mirroring and appending our data set somehow fundamentally flawed? We are looking at other approaches such as using a low pass filter as well.

Improve this question
$\endgroup$
3
  • $\begingroup$ What about slow (standard) Fourier transform. $\endgroup$
    – user88
    Commented Jul 22, 2010 at 11:45
  • $\begingroup$ Are these differentially corrected GPS measurements of plate motion, by any chance? $\endgroup$
    – whuber ♦
    Commented Jun 7, 2012 at 20:32
  • $\begingroup$ It was actually movements of a tunnel whilst construction work was going on around it. We expected the movement to very roughly follow an S curve over the period of monitoring. $\endgroup$
    – Ian Turner
    Commented Jun 29, 2012 at 13:53

6 Answers 6

Reset to default
12
$\begingroup$

It sounds dodgy to me as the trend estimate will be biased near the point where you splice on the false data. An alternative approach is a nonparametric regression smoother such as loess or splines.

Improve this answer
$\endgroup$
10
$\begingroup$

If you want to filter the long term trend out using signal processing, why not just use a low-pass?

The simplest thing I can think of would be an exponential moving average.

Improve this answer
$\endgroup$
2
  • $\begingroup$ We had a look at it. Worked okay but in this case the noise still seemed to be a bit too strong and if we changed the parameters to even out the distributions enough it appeared that the trend was damped down too much. Maybe in this case there just is no solution to the data and it is just a bit too noisy. $\endgroup$
    – Ian Turner
    Commented Jul 24, 2010 at 17:10
  • 2
    $\begingroup$ Exponentially weighted moving averages are a special case of a kernel smoother (assuming you used a 2-sided MA rather than 1-sided). Better estimates that are generalizations of this are loess or splines -- see my answer. $\endgroup$
    – Rob Hyndman
    Commented Jul 25, 2010 at 9:15
8
$\begingroup$

You could use the (fast :) ) discrete wavelet transform. The package wavethresh under R will do all the work. Anyway, I like the solution of @James because it is simple and seems to go straigh to the point.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Agreed; wavelets are excellent for picking out non-stationary behavior in high amounts of noise. You do have to be careful with the DWT, though. It's not rotation-invariant (although there are modifications of the DWT that are, see e.g. Percival and Walden 2000), so you can lose sharp transients depending on the starting point for your data. Also, most implementations of the DWT do implicit circularization of the data, so you'd still need to control for that. $\endgroup$
    – Rich
    Commented Aug 3, 2010 at 19:57
  • $\begingroup$ If my memory is good, package wavethresh contains translation invariant denoising (my reference was Coifman 1995) (Note that you talked about rotation, arn't we talking about temporal signals?). $\endgroup$
    – robin girard
    Commented Aug 3, 2010 at 20:53
  • $\begingroup$ are you talking about MODWT (Maximum Overlap Discrete Wavelet Transform) ? $\endgroup$
    – RockScience
    Commented Feb 9, 2011 at 6:59
  • $\begingroup$ @fRed: nop, here is the paper, Coifman and Donoho: citeseerx.ist.psu.edu/viewdoc/… $\endgroup$
    – robin girard
    Commented Feb 9, 2011 at 7:17
7
$\begingroup$

I think you can get some distortion on the pasting point as not all the underlying waves will connect very well.

I would suggest using a Hilbert Huang transform for this. Just do the split into intrinsic mode functions and see what is left over as residue when calculating them.

Improve this answer
$\endgroup$
4
$\begingroup$

Most of the time when I hear "long-term trend", I think of long-term upward trends or long-term downward trends, neither one of which is properly captured by a Fourier transform. Such one-way trends are better analyzed by using linear regression. (Fourier transforms and periodograms are more appropriate for things that go up and down).

Linear regression is easy to do in most spreadsheets. (a) Display equations for regression lines (b) Creating XY Scattergraphs with Spreadsheets

Linear regression tries to approximate your data with a straight line. Fourier transforms try to approximate your data with a few sine-waves added together. There are other techniques ("non-linear regression") that try to approximate your data to polynomials or other shapes.

Improve this answer
$\endgroup$
2
$\begingroup$

The Fourier transform assumes wide sense signal stationarity and linear time invariance (LTI). While it is robust to some violation of these conditions, I don't really think it is appropriate for analysis of trends due to the assumption of stationarity, i.e. you are trying to measure something which violates one of the FFTs basic assumptions.

I would agree with the posters above; mirroring your data and adding the mirrored data to the end of your time-series is dodgy. I would suggest that fitting a linear regression model with a time trend as mentioned above is probably more appropriate.

If you were looking to examine periodicity, you could remove the trend by high pass filtering and performing a Fourier analysis. If the trend remains visible after filtering you could subtract a fitted linear regression line from the original signal prior to FFT.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.