3
$\begingroup$

I have data like the following image, where the x-axis is the absolute elapsed time in hours (think calendar days; this plot goes over ~2.5 years), and the y-axis is the manually entered uptime of a machine in hours.

For my data processing, I need a very robust way of smoothing the data. The image below shows a black, dashed line, which is the best I can get with a "normal" GAM.

I am thinking of a GAM that minimizes the median residual or something. My ideal goal is to obtain something close to the manually drawn red line.

I have tried many smoothers (cubic polynomial, loess, GAM, Theil-Sen linear regression), but every one fails in some special cases of the hundreds of graphs I have.

What are other robust smoothers that I could apply here? Or are there other techniques I have completely missed so far?

enter image description here

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ As a machine can't have negative uptime (can it in your context have 0 uptime?), a GAM (or any model) with a Gaussian distribution for the response conditional upon the covariates is unlikely to work well - you can see partly why in your figure; at well before zero on the x-axis, the response is predicted to be negative. Also, a GAM in the sense that I understand you use of the term isn't minimising the "median residual". If this was a Gaussian GAM then it is minimising the penalised sum of squared errors, so is just like a linear regression plus the extra penalty on wiggliness. $\endgroup$
    – Gavin Simpson
    Commented Jun 8, 2017 at 15:33
  • 2
    $\begingroup$ You will definitely want to include the known constraints: the curve must pass through the origin, and must be non-decreasing (i.e. $y_0=0$ and $y_{i+1}\geq{y_i}$). $\endgroup$
    – GeoMatt22
    Commented Jun 8, 2017 at 16:11
  • 1
    $\begingroup$ It might sound obvious but if you have constraints on the values your response transforming your data in a domain where this restrictions are automatically met is a viable option. For instance why not take $log_{10}$ of your data? The back-transformation will immediately force the data to be positive. Similarly as @GeoMatt22 mentioned if you have a non-decreasing trend in your data, isotonic regression seems an obvious thing to try (Google smooth isotonic regression). BTW the grey dashed line "seems mostly OK" to me. $\endgroup$
    – usεr11852
    Commented Jul 1, 2017 at 12:21

1 Answer 1

Reset to default
1
$\begingroup$

It looks like you mainly want to reject data at time 3000 and 28000. If you're going to believe all your data points, the dashed line looks reasonable. But you want to reject or replace the outliers, to achieve robust smoothing.

One specific approach to answer the question is to make a pass through the data in time order using a spike filter. That's very simple - see an explanation and pseudocode at http://gregstanleyandassociates.com/whitepapers/FaultDiagnosis/Filtering/Spike-Filter/spike-filter.htm That would reject the first obviously bad point,and at least flag the last point as questionable, depending on the maximum count parameter. That particular approach is a filter, not smoothing (at any point, only looking backwards, not ahead), so it wouldn't be optimal for your case, where at any time (except the beginning and end) you could look backwards or forwards. That spike filter would carry over only past values instead of considering both past and future, which would distort the curve slightly.

It would be better to either delete the outlier or replace it with a locally-interpolated value. You could still do this with one pass in time order. I'd modify the above filter to just mark the data until it was resolved as a spike or a longer-lasting change (resolved after the time period specified by the count parameter). When a spike is detected, then either delete those marked points or fill in with interpolated values rather than a previous value. The choice of deleting the marked points or filling in with a locally interpolated value is could be based on whether or not you need values at fixed intervals, or also whether you want to have one global function or something based on nearby points, as in Savitzky-Golay smoothing.

Savitzky-Golay smoothing is an extremely efficient implementation of least squares smoothing over a sliding time window (a convolution over that data) that comes down to just multiplying the data in each time window by fixed constants. You can fit values, derivatives, second derivatives, and higher. There's a good article in Wikipedia, although it's referred to as Savitzky-Golay filter.

Savitzky-Golay smoothing is specifically not for robust smoothing. It was derived based on a least squares fit of data within a time window of nearby points. Least squares answers are inherently not robust, because the squared objective function gives too much weight to outliers. But, in conjunction with the above spike detection logic, it might be used in local interpolation once the spikes are removed, and might be used for all data points after outlier removal.

Since your data doesn't appear to be regularly sampled (fixed sample time) anyway, you couldn't use Savitzky-Golay. So you would just delete the points marked as outliers. The above is included for others with similar questions with regularly-sampled data.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.