Given a data set, I want to divide it in two different sets if I see that part of the data misbehaves. For example, in the figure you can clearly see that something is happening before the vertical red line. So I was able to determine the lines by using the Jenks natural breaks optimization by using 2 groups. The problem is that if the data is better behaved (say there are no such big jumps and the data is smoother, like an exponential function), Jenks algorithm still divides it in two. So I have two questions:

  1. Is there a way to automatically (no thresholds, no parameters set by the analyst) identify if there's an anomaly that would require the use of Jenks algorithm?
  2. Is there a way to quantify the anomaly? I would like to have a factor between 0 to 1, the latter corresponding to a smooth function and 0 corresponding to a sort of Heaviside anomaly behavior. Of course the figure shows something in between, so it would be closer to 0, say 0.3.

enter image description here

  • 1
    $\begingroup$ Could you please explain the nature of this graphic and what it represents? What exactly do you mean by an "anomaly" or "misbehaving"? Since the Jenks' method is for univariate data and your plot seems to show bivariate data, could you give us some description of your data so we can understand what you're trying to do? $\endgroup$ – whuber Aug 17 '15 at 21:54
  • $\begingroup$ It's univariate data, and that's the reason I don't show the x-axis because it would be 1, 2, 3, and so on. By anomaly I mean that the behavior is departing from an exponential decay. Notice that the y values are discrete. The ordinates refer to chemical reaction values. $\endgroup$ – aaragon Aug 17 '15 at 22:03
  • 1
    $\begingroup$ Your reference to "exponential decay" indicates you are thinking of these as bivariate data: you have described the relationship between the y values and x values. Therefore I'm still quite puzzled about what you're trying to ask, since the text refers to strictly univariate clustering methods. And once again: what do you mean by "misbehave" and "anomaly"? $\endgroup$ – whuber Aug 17 '15 at 23:20
  • $\begingroup$ There is only a single variable in data as the abscissa is irrelevant. Think of the data as a sorted 1D data, arranged such that you have a moronically decreasing function. If the function is well behaved, there shouldn't be any jumps as I show in this figure. These jumps is what I refer as anomaly. $\endgroup$ – aaragon Aug 18 '15 at 4:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.