0
$\begingroup$

I have a numpy array full of customer spending data:

x = np.array([5000,5500,6250,4800,3950,5800,5500,800,1200,900,500,400,300,200,3100])

Above, you can see that before index 7 the customer spends much more money than he does after index 7. I am looking to find an abnormality such as index 7 by looking sequentially at the data and want to identify if the split data set has a significant change (i.e. spending habits have changed or remained the same after the first abnormality).

So, 800 would be detected and there would be two lists:

l1 = [5000,5500,6250,4800,3950,5800,5500] 
l2 = [1200,900,500,400,300,200,3100] 

Here, a similarity measure needs to be upon comparison such to see the similarity or difference of l1 and l2.

Are there any useful sklearn functions for this? I know I can look at means, rolling std's etc, and set thresholds for this but was looking for a more statistical approach possibly something in a more statistical/machine learning python library.

$\endgroup$

migrated from stackoverflow.com Jul 1 '17 at 14:19

This question came from our site for professional and enthusiast programmers.

  • $\begingroup$ Research a bit about "clustering" in general and KMeans especially, I think it's what you need to intelligently devide the user behaviour into discrete clusters $\endgroup$ – Ofer Sadan Jun 30 '17 at 18:28
  • $\begingroup$ I know about Means and Affinity propagation, but this relies on a sequential aspect. I don't want to just set k = 2 or k = 3 as there can be an out of order arrangement of elements across these clusters. I.e. p_1,p_2,...,p_k all need to be together then p_k+1, p_k+2,...,p_n need to be together $\endgroup$ – Mike El Jackson Jun 30 '17 at 18:30
  • $\begingroup$ A possible approach is to find inflection point by finding the concavity changes (look at the second derivative). $\endgroup$ – Anis Nouri Jul 1 '17 at 15:08
0
$\begingroup$

Regarding the list you passed, I don't really understand why 3100is in l2 and not in l1.
However, you find a way to split you list by comparing new numbers. For example, you can use the gradient, when there is a gradient jump is where you have to split. Or you can compare each number of the list to mean of the previous numbers. Without a clustering method, I would begin with something like this :

l1 = l2 = []
x = np.array([5000,5500,6250,4800,3950,5800,5500,800,
1200,900,500,400,300,200,3100])
for i, k in enumerate(x):
    if len(l1) > 2:
        m1 = np.mean(l1)
        if np.abs(m1 -k)/m1 > 0.5:
            index = i
            break
    l1.append(k)
l2 = x[index:]
In [11]: l1, l2
Out[11]:
([5000, 5500, 6250, 4800, 3950, 5800, 5500],
 array([ 800, 1200,  900,  500,  400,  300,  200, 3100]))
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy