4
$\begingroup$

I have a noisy readout of a curve that is monotonically increasing or decreasing for a narrow range of points and then quickly saturates. I don't know exactly where the saturation point is, but from the underlying (biological) process I know that it exists and once it's reached, strong deviations from a flat line are very likely to be outliers.

I am looking for the best way to smooth these data. Something like this (the data is actually not for time series, but it's a good analogy):

Example of data and a curve I'd like to fit

So far I have tried using:

  • the Hampel filter
  • loess regression (including its robust version, family="symmetric" in R)
  • a combination of both

The problem with loess is that since the non-saturated bit is so small, loess tends to "underfit" it, "dragging" the trendline up or down towards the saturation point. The problem with the Hampel filter is that while it does a great job in filtering out strong outliers, it's not that great for smoothing as such, i.e., polishing out small deviations from the trendline.

It seems like the best thing would be to find the saturation point and then fit two separate models "before" and "after" it. This would be straightforward without the noise, but I'm not sure how to do it in a noisy setting. I'm also open to any other suggestions.

This is part of a large project in R, so if a suggested algorithm has an R implementation this would be particularly great.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Depending on role of this analysis in the larger project, you could do many things. Are you trying only to find outliers (and do not need the regression function(s))?

If so, you could consider changepoint analysis, which looks for changepoints in data that can be viewed as a time series. An implementation of several common algorithms can be found in the R package changepoint, but the gist for each of them is that the algorithm will find points where the mean (or variance or both) of your data change.

An issue with this approach is determining which time periods are or are not aberrant. If it is reasonable to assume that the majority of observations are made during the period of saturation, then one possible way around this issue is to assume that other periods that are not outliers must have means (or variances or both) similar to that of the mean of the longest number of observations.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for your suggestion, I will certainly have a look at changepoint. I need to smoothen the function, but not necessarily find a formal dependence y=f(x), as all I need to be able to do is get back to each of these discrete time points and find a smoothened readout for it. In a slightly different setting, where the period of saturation was shorter (or at least, not so much longer) than that of monotonous change, I used loess regression without a problem, but here it doesn't seem to work that well. $\endgroup$
    – a11msp
    Jun 17, 2013 at 15:37
1
$\begingroup$

If your data happens to be multivariate time series (or something that behaves similarly), maybe Multivariate Curve Resolution could help.

Literature: A. de Juan, M. Mäder, M. Martínez, and R. Tauler: Combining hard- and soft-modelling to solve kinetic problems, Chemometrics and Intelligent Laboratory Systems, 2000, 54, 123-141. DOI: 10.1016/S0169-7439(00)00112-X

In R, package ALS gives an implementation, and it is also discussed in the UseR! Chemometrics book.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.