2
$\begingroup$

To construct a plot, I'm looking for an algorithm which can handle inf and (very) negative values.

If I have infinity values everything is a line but not infinities.

Example

57474
65637
35252
17938
inf
56268
12341
-200000
45884
inf
36561
-7000

There are not so many anomalies like in the example, but I just want to show the reference data range

The reason I need smoothing is to avoid zig-zag enter image description here

But something like this

Example from the internet

So I need an algorithm which can show a pretty line and ignore single outliers

$\endgroup$
8
  • $\begingroup$ You put "Smoothing algorithm" into the title. But what is it you are trying to do with these data? $\endgroup$ – Fato39 Sep 14 '20 at 9:27
  • $\begingroup$ Can you, please, include these details into your question? Give an example of what kind of visualization you are trying to do (e.g. a bar plot). It will help people provide answers that will match best to your use case. As it is, your question is very broad and difficult to answer. $\endgroup$ – Fato39 Sep 14 '20 at 10:07
  • $\begingroup$ @Fato39 done. Thank you for your attention. Btw which alhorithm should I use? $\endgroup$ – Noo.B Sep 14 '20 at 10:59
  • $\begingroup$ If you already know it’s inf and negative values, can you just filter it out? Why do you need an algoritm? $\endgroup$ – forecaster Sep 14 '20 at 11:04
  • $\begingroup$ @forecaster sorry that I was inaccurate again. The question is updated $\endgroup$ – Noo.B Sep 14 '20 at 11:10
4
$\begingroup$

A classical solution is to apply a robust local smoother.

In his book EDA (Addison-Wesley 1977), John Tukey lays out principles and procedures based primarily on running local medians. When a median of an odd number $2k+1$ of data is computed, up to $k$ of those values may be extreme yet the median will be finite and well-defined.

Handling infinities is still problematic. If the infinities are intended to represent truly large numbers, then keep them as-is; but otherwise, they function as missing values. Handle them simply by leaving them out of the time series when smoothing it.

Below is an example using Tukey's 3RSSH smoother. This notation refers to performing running medians of three data repeatedly until nothing changes, then performing two "splitting" operations to handle strings of repeated values, followed by a "Hanning" step which performs a weighted running mean within a three-element window (thereby averaging out some of the blocky jumps that can be created by the running medians). It's efficient and in practice works reasonably well for exploring data.

To see how well any smoother works, explore the residuals: these, by definition, are the differences between the data and their smoothed values. When residuals show local trends, you can go further by smoothing the residuals. Tukey provides the details and many worked examples.

Figure

This dataset was created to look like the first plot in the question. Then the values were replaced by "infinity" at ten randomly selected times. These couldn't be plotted at the left, so they show up as gaps in the graph. At the right, their positions are filled with orange crosses to confirm these data haven't been overlooked.

These computations are partially implemented in the base R program within its smooth function. Hanning is straightforward to code as a convolution. Additional code is needed to identify and skip over the infinite values. Here is the resulting implementation:

s <- function(x) {
  hann <- function(x) {
    n <- length(x)
    x <- c(x[1], x, x[n])
    convolve(x, c(1,2,1)/4, type="filter")
  }
  #
  # Smooth the non-infinite, non-missing values of `x`.
  #
  y <- x
  i <- which(!is.infinite(x) & !is.na(x))
  y[i] <- hann(smooth(x[i], "3RSS"))
  y
}

Long ago, I shared an Excel macro to implement all of Tukey's smoothers. (It goes well beyond the R implementation.) It probably needs porting to newer versions of Excel, but the underlying code ought to work still. Google 'Tukey EDA Excel' to find it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.