Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 8 down vote favorite

This may sound very basic, but I have this problem: I've got a queue of data with a window size of 300. New data is added at one end, old values are removed from the other end.

I expect the queue data to stay more or less consistent, e.g.: 10,12,15,10,20, then start to rise sharply: 15,10,20,22,25,26,28,30,32... all the way up to 150 or so. There the data may fluctuate a bit, then it would go down with a similar slope (120,118,116,115...) all the way down to 20 or so.

I'm trying to identify the turning point in this data series programmatically, but my code detects peaks more often than I'd like to. How can I identify when the graph is rising, when it has reached a definitive turning point and when it starts to fall? Should I try to look at the rate of change of the rate of change?

share|improve this question
If you have a reasonable sense of the slope andthe height of the peak, could you take account of both height and how long it's been climbing, as well as the current slope? How much prior data on peaks do you have, and could you use those to fit a model? – Karl Oct 18 '11 at 3:10
2  
Whatever the method, you're going to have to properly define what is a peak and what is a plateau. Are two consecutive 'high' values a peak or a plateau? How about five? Some methods may not appear to need this information, but then this is hidden behind the scenes. As ever so often, it comes down to properly defining your problem, and the optimizing your parameters (and/or algorithm) for it. – Nick Sabbe Oct 18 '11 at 9:41

2 Answers

up vote 2 down vote accepted

If you know that this is the exact pattern to expect, then you can look for this exact pattern, but then you will miss other patterns. So. If you know that the peak will be 150, then you could look for 2 or 3 or 4 or (however many) consecutive values of 150. But you say "or so" - how big is the "or so"? Perhaps the peak is defined as "3 consecutive values over 130" or maybe it's "3 out of 5 consecutive values over 140". That's for you to decide.

On the other hand, if you are just looking for some general program to detect peaks - well, that's been looked at. There are a bunch of smoothing methods (e.g. loess, splines of various sorts, moving averages etc.). Not a field I'm expert in, but there's lots of literature on this.

share|improve this answer
up vote 3 down vote

Look into SiZer (SIgnificant ZERo crossings... or slopes, I don't remember), although arguably it is more of a cross-sectional than time-series tool. The idea there is to smooth the data at different bandwidths (varying by some three orders of magnitude), and apply some local tests to see whether the slope of a local regression is significantly positive or negative (or undecided). It produces a convincing picture that would aid you in determining which features are there. (I am surprised there is no R implementation, only Matlab.)

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.