4
$\begingroup$

I currently have many sets of data that display more or less the trend in the image, which may be due to abnormalities of the data source. The series "splits" into two different trends, with one growing exponentially and the other scattered and growing much slower. Since only the first trend is desired, those that belong to the latter trend needs to be removed. I'm looking for a method that can be used to remove the data points that isn't growing as per the trend.

The data are series of time-value Pairs. The following plot shows one of the series, with the horizontal axis being the time axis, and the vertical axis being the value recorded at that time.

Sample Data:

    06:35:00    342.0
    06:35:44    332.0
    06:35:47    337.0
    06:40:53    387.0
    06:45:07    383.0
    06:45:10    369.0
    06:46:38    395.0
    06:51:44    384.0
    06:51:45    383.0
    06:52:57    381.0
    06:53:55    384.0
    06:57:38    384.0

I have though out slicing the data into small intervals, cluster them by k-means into two groups then compare the mean. But I'm looking for better methods that may have fewer effects to the first section of the data.

Data Trend

$\endgroup$
4
  • $\begingroup$ please describe your data .. $\endgroup$
    – IrishStat
    Jul 17, 2017 at 22:28
  • $\begingroup$ Apologies. I've added description and sample data. $\endgroup$
    – Kev W.
    Jul 17, 2017 at 22:33
  • 2
    $\begingroup$ +1 Interesting question. If you are familiar with robust smoothers, you might try a few to see whether they succeed in following one or the other trajectory. (IWLS methods often do.) When that's the case, you can identify the "outliers" and then it remains only to decide whether the "outliers" are the portion you want to retain or the portion you want to throw away (which would be based on the trend estimates for each option). Repeat once or twice to polish the result. You will be in trouble with this method if the two trajectories tend to have equal numbers of points. $\endgroup$
    – whuber
    Jul 18, 2017 at 13:06
  • $\begingroup$ @whuber Thanks for the suggestion! I'll look into this approach. $\endgroup$
    – Kev W.
    Jul 18, 2017 at 22:48

1 Answer 1

-1
$\begingroup$

Since you have unequal intervals between readings time series methods are inapplicable. BUT I would consider using an average reading for a fixed interval of time say 60 second intervals and build/identify a useful model. In this way you can use times series methods with intervention detection to sort out memory effects , trends , intercept changes , pulses etc in order to create an equation which is robust to the anomalies. Sometimes when faced with a possibly unsolvable problem ( not only temporal effects but an exogenous unknown causal series effect ) you have to be innovative . Post that time series and if I have time available I will "look" at it with my microscope.

Response to @whuber correctly reflecting that the average was a weighted sum of perhaps two distributions thus injecting an effect that needs to be identified. The temporal portion/effect can be critical to assessing/detecting the inconsistency present in the next time bucket and must be addressed first , in my opinion. Statistics is the practice of identifying regularity for the express purpose of identify/unveiling irregularity, if any. As Bacon said (and I paraphrase ) "the deviations of nature aid the understanding of the true laws of nature "

What I had in mind is that obtaining averages for a "bucket of time" would enable the following. Assume that we have 100 time buckets and that unknown unusual activity occurred at periods 61,62 and 63 . Now if we use 60 values to predict the 61st an anomaly will be detected even though the value foe bucket 61 was the average of good and irregular observations. We now know that period 61 needs to be put under a microscope and closely evaluated. The actual readings for period 61 can be compared to the prediction for period 61 to address homogeneity and thus we then can isolate the reasonable from the unreasonable. If we find the actual readings to be homogeneous then we have detected a true time-related anomaly AND not the presence of two separate distributions.

If the bucket value for 61 is not identified as being a pulse given the first 60, we move on to build/identify a useful model using 61 values to perhaps identify the 62nd value as being an outlier/pulse reflecting the "possible" heterogeneous values used to construct the 62nd value of the 100.

$\endgroup$
1
  • 1
    $\begingroup$ Averaging the readings would mix the very two phenomena the OP is trying to tease apart! I suspect you might have focused on the temporal nature of the data without reading the question. $\endgroup$
    – whuber
    Jul 18, 2017 at 13:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.