# Finding change in spending habits

I have a numpy array full of customer spending data:

x = np.array([5000,5500,6250,4800,3950,5800,5500,800,1200,900,500,400,300,200,3100])


Above, you can see that before index 7 the customer spends much more money than he does after index 7. I am looking to find an abnormality such as index 7 by looking sequentially at the data and want to identify if the split data set has a significant change (i.e. spending habits have changed or remained the same after the first abnormality).

So, 800 would be detected and there would be two lists:

l1 = [5000,5500,6250,4800,3950,5800,5500]
l2 = [1200,900,500,400,300,200,3100]


Here, a similarity measure needs to be upon comparison such to see the similarity or difference of l1 and l2.

Are there any useful sklearn functions for this? I know I can look at means, rolling std's etc, and set thresholds for this but was looking for a more statistical approach possibly something in a more statistical/machine learning python library.

## migrated from stackoverflow.comJul 1 '17 at 14:19

This question came from our site for professional and enthusiast programmers.

• Research a bit about "clustering" in general and KMeans especially, I think it's what you need to intelligently devide the user behaviour into discrete clusters – Ofer Sadan Jun 30 '17 at 18:28
• I know about Means and Affinity propagation, but this relies on a sequential aspect. I don't want to just set k = 2 or k = 3 as there can be an out of order arrangement of elements across these clusters. I.e. p_1,p_2,...,p_k all need to be together then p_k+1, p_k+2,...,p_n need to be together – Mike El Jackson Jun 30 '17 at 18:30
• A possible approach is to find inflection point by finding the concavity changes (look at the second derivative). – Anis Nouri Jul 1 '17 at 15:08

Regarding the list you passed, I don't really understand why 3100is in l2 and not in l1.
However, you find a way to split you list by comparing new numbers. For example, you can use the gradient, when there is a gradient jump is where you have to split. Or you can compare each number of the list to mean of the previous numbers. Without a clustering method, I would begin with something like this :

l1 = l2 = []
x = np.array([5000,5500,6250,4800,3950,5800,5500,800,
1200,900,500,400,300,200,3100])
for i, k in enumerate(x):
if len(l1) > 2:
m1 = np.mean(l1)
if np.abs(m1 -k)/m1 > 0.5:
index = i
break
l1.append(k)
l2 = x[index:]
In : l1, l2
Out:
([5000, 5500, 6250, 4800, 3950, 5800, 5500],
array([ 800, 1200,  900,  500,  400,  300,  200, 3100]))