Remove Outlying Data with a Different Trend

Question

I currently have many sets of data that display more or less the trend in the image, which may be due to abnormalities of the data source. The series "splits" into two different trends, with one growing exponentially and the other scattered and growing much slower. Since only the first trend is desired, those that belong to the latter trend needs to be removed. I'm looking for a method that can be used to remove the data points that isn't growing as per the trend.

The data are series of time-value Pairs. The following plot shows one of the series, with the horizontal axis being the time axis, and the vertical axis being the value recorded at that time.

Sample Data:

    06:35:00    342.0
    06:35:44    332.0
    06:35:47    337.0
    06:40:53    387.0
    06:45:07    383.0
    06:45:10    369.0
    06:46:38    395.0
    06:51:44    384.0
    06:51:45    383.0
    06:52:57    381.0
    06:53:55    384.0
    06:57:38    384.0

I have though out slicing the data into small intervals, cluster them by k-means into two groups then compare the mean. But I'm looking for better methods that may have fewer effects to the first section of the data.

Answer 1

Since you have unequal intervals between readings time series methods are inapplicable. BUT I would consider using an average reading for a fixed interval of time say 60 second intervals and build/identify a useful model. In this way you can use times series methods with intervention detection to sort out memory effects , trends , intercept changes , pulses etc in order to create an equation which is robust to the anomalies. Sometimes when faced with a possibly unsolvable problem ( not only temporal effects but an exogenous unknown causal series effect ) you have to be innovative . Post that time series and if I have time available I will "look" at it with my microscope.

Response to @whuber correctly reflecting that the average was a weighted sum of perhaps two distributions thus injecting an effect that needs to be identified. The temporal portion/effect can be critical to assessing/detecting the inconsistency present in the next time bucket and must be addressed first , in my opinion. Statistics is the practice of identifying regularity for the express purpose of identify/unveiling irregularity, if any. As Bacon said (and I paraphrase ) "the deviations of nature aid the understanding of the true laws of nature "

What I had in mind is that obtaining averages for a "bucket of time" would enable the following. Assume that we have 100 time buckets and that unknown unusual activity occurred at periods 61,62 and 63 . Now if we use 60 values to predict the 61st an anomaly will be detected even though the value foe bucket 61 was the average of good and irregular observations. We now know that period 61 needs to be put under a microscope and closely evaluated. The actual readings for period 61 can be compared to the prediction for period 61 to address homogeneity and thus we then can isolate the reasonable from the unreasonable. If we find the actual readings to be homogeneous then we have detected a true time-related anomaly AND not the presence of two separate distributions.

If the bucket value for 61 is not identified as being a pulse given the first 60, we move on to build/identify a useful model using 61 values to perhaps identify the 62nd value as being an outlier/pulse reflecting the "possible" heterogeneous values used to construct the 62nd value of the 100.