1
$\begingroup$

I am working on my thesis but I am stuck on the last task to solve.

STEP 1) I have a dataset composed of three variables:

  • fidelity_card_ID: fidelity card associated to the purchases
  • shopping_date: day when the purchases were made
  • cluster: express the pattern of this shopping visit

Examples of clusters description are: shopping for clothes, shopping for housecleaning products, shopping for a meal, shopping for weekly grocery, etc.

STEP 2) Each fidality_card_ID has a unique profile in terms of clusters composition.
For example, 100% of shopping visits made by fidelity_card_ID == 1 are clustered as "shopping for clothes". On the other hand, there is fidelity_card_ID == 2 which 99% of shopping visits were clustered as "shopping for housecleaning products" and there is 1% of shopping visits clustered as "shopping for a meal".

Question

STEP 3) What is the correct approach to develop a model to classify/predict/detect for each fidelity_card_ID those shopping vists that do not belong to the recurrent pattern of that specific fidelity_card_ID?
For example, this model should "highlight" the 1% of shopping visits clustered as "shopping for meal" of fidelity_card_ID == 2 and it should "not highlight" any shopping visit made by fidelity_card_ID == 1 because they all belongs in its recurrent pattern.

One of the possible object is understand whether there are several different people sharing the same fidelity_card_ID.

Improve this question
$\endgroup$
1
  • $\begingroup$ I can't find the correct terminology but the concept is that I need a model that learn the normal behaviour of each ID and identify anomalies in his behaviour. This must be "looped" for each ID in the dataset. $\endgroup$
    – Seymour
    Commented Feb 5, 2018 at 11:47

1 Answer 1

Reset to default
2
$\begingroup$

You are looking at an unsupervised learning problem, i.e. your transactions do not have "regular" or "irregular" activity labels. Regularity is customer dependent, you can try to derive customer specific regularity features, e.g. the most frequent category for that customer (and whether or not a new activity is deviant from that)given day of the week, location of the customer, etc. and then label some of your data (semi-supervised, just because labelling all may not be feasible) and fit a single classifier. There will not be an easy shortcut here I am afraid.

Depending on your dataset, you can carry out novelty & outlier detection.

Or you can look at one-class supervised learning.

I am not going into more detail, there are plenty of threads on this website discussing these two.

Improve this answer
$\endgroup$
5
  • $\begingroup$ in the case of one-class supervised learning,does it imply that I must learn a classifier for each fidelity_card_ID? $\endgroup$
    – Seymour
    Commented Feb 5, 2018 at 12:24
  • $\begingroup$ Because what confuses me the most is that the "regular" or "irregular" activity labels is different for each of the thousands of fidelity_card_ID $\endgroup$
    – Seymour
    Commented Feb 5, 2018 at 12:27
  • $\begingroup$ Yes, not just for that though, if you want to condition "regular" vs"irregular" activity, you would have to either manually train N models for N ID's or you just input "ID" as a categorical predictor variable. It is true that you will be introducing a cat variable with high cardinality and will therefore need a lot of data per ID. $\endgroup$
    – Zhubarb
    Commented Feb 5, 2018 at 12:33
  • $\begingroup$ What alternative can you suggest to handle this issue? Because running one classifier for each ID is not feasible. In my mind, I would need some kind of "density-based" detector/predictor which define as "irregular" those shopping visits that do not belong to the normal behaviour of that specific customer. Any idea? $\endgroup$
    – Seymour
    Commented Feb 5, 2018 at 13:19
  • $\begingroup$ Regularity is customer dependent, you can try to derive customer specific regularity features, e.g. the most frequent category for that customer (and wehther or not a new activity is deviant from that)given day of the week, location of the customer, etc. and then label some of your data (semi-supervised, just because labelling all may not be feasible) and fit a single classifier. There will not be an easy shortcut here I am afraid. $\endgroup$
    – Zhubarb
    Commented Feb 5, 2018 at 13:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.