I have a numpy array full of customer spending data:

x = np.array([5000,5500,6250,4800,3950,5800,5500,800,1200,900,500,400,300,200,3100])

Above, you can see that before index 7 the customer spends much more money than he does after index 7. I am looking to find an abnormality such as index 7 by looking sequentially at the data and want to identify if the split data set has a significant change (i.e. spending habits have changed or remained the same after the first abnormality).

So, 800 would be detected and there would be two lists:

l1 = [5000,5500,6250,4800,3950,5800,5500] 
l2 = [1200,900,500,400,300,200,3100] 

Here, a similarity measure needs to be upon comparison such to see the similarity or difference of l1 and l2.

Are there any useful sklearn functions for this? I know I can look at means, rolling std's etc, and set thresholds for this but was looking for a more statistical approach possibly something in a more statistical/machine learning python library.

  • $\begingroup$ Research a bit about "clustering" in general and KMeans especially, I think it's what you need to intelligently devide the user behaviour into discrete clusters $\endgroup$
    – Ofer Sadan
    Jun 30, 2017 at 18:28
  • $\begingroup$ I know about Means and Affinity propagation, but this relies on a sequential aspect. I don't want to just set k = 2 or k = 3 as there can be an out of order arrangement of elements across these clusters. I.e. p_1,p_2,...,p_k all need to be together then p_k+1, p_k+2,...,p_n need to be together $\endgroup$
    – Mike El Jackson
    Jun 30, 2017 at 18:30
  • $\begingroup$ A possible approach is to find inflection point by finding the concavity changes (look at the second derivative). $\endgroup$
    – Anis Nouri
    Jul 1, 2017 at 15:08

1 Answer 1


Regarding the list you passed, I don't really understand why 3100is in l2 and not in l1.
However, you find a way to split you list by comparing new numbers. For example, you can use the gradient, when there is a gradient jump is where you have to split. Or you can compare each number of the list to mean of the previous numbers. Without a clustering method, I would begin with something like this :

l1 = l2 = []
x = np.array([5000,5500,6250,4800,3950,5800,5500,800,
for i, k in enumerate(x):
    if len(l1) > 2:
        m1 = np.mean(l1)
        if np.abs(m1 -k)/m1 > 0.5:
            index = i
l2 = x[index:]
In [11]: l1, l2
([5000, 5500, 6250, 4800, 3950, 5800, 5500],
 array([ 800, 1200,  900,  500,  400,  300,  200, 3100]))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.