Finding change in spending habits

Question

I have a numpy array full of customer spending data:

x = np.array([5000,5500,6250,4800,3950,5800,5500,800,1200,900,500,400,300,200,3100])

Above, you can see that before index 7 the customer spends much more money than he does after index 7. I am looking to find an abnormality such as index 7 by looking sequentially at the data and want to identify if the split data set has a significant change (i.e. spending habits have changed or remained the same after the first abnormality).

So, 800 would be detected and there would be two lists:

l1 = [5000,5500,6250,4800,3950,5800,5500] 
l2 = [1200,900,500,400,300,200,3100] 

Here, a similarity measure needs to be upon comparison such to see the similarity or difference of l1 and l2.

Are there any useful sklearn functions for this? I know I can look at means, rolling std's etc, and set thresholds for this but was looking for a more statistical approach possibly something in a more statistical/machine learning python library.

Answer 1

Regarding the list you passed, I don't really understand why 3100is in l2 and not in l1.
However, you find a way to split you list by comparing new numbers. For example, you can use the gradient, when there is a gradient jump is where you have to split. Or you can compare each number of the list to mean of the previous numbers. Without a clustering method, I would begin with something like this :

l1 = l2 = []
x = np.array([5000,5500,6250,4800,3950,5800,5500,800,
1200,900,500,400,300,200,3100])
for i, k in enumerate(x):
    if len(l1) > 2:
        m1 = np.mean(l1)
        if np.abs(m1 -k)/m1 > 0.5:
            index = i
            break
    l1.append(k)
l2 = x[index:]
In [11]: l1, l2
Out[11]:
([5000, 5500, 6250, 4800, 3950, 5800, 5500],
 array([ 800, 1200,  900,  500,  400,  300,  200, 3100]))