Firstly, I'm not sure if this is the right instance of StackOverflow to post on so feel free to ask me to put it elsewhere.

I am exploring the concepts of clustering and "unsupervised" learning for the first time.

I can't say too much about the data, but "risky" is in the context of the users carrying out transactions (on behalf of the company) that might be construed as exposing the organisation to unnecessary risk.

So I have two datasets:

  • a table of transactions made by users (so a single user will have 1 or more transactions involving money)
  • a table of users with a certain percentage who have been flagged as showcasing "dangerous" behaviour - this is a smallish percentage so far (around 4%) so I understand if these were classes, it's an "unbalanced class" problem

For the users, a percentage have been flagged as "risky" but there are others yet to be identified/remain unflagged.

The thing I'm trying to do is to figure out:

  • What about their transactional behaviour caused certain users to be flagged?
  • Can I identify a few other unflagged users?

For the first point, I feel that I might have to do some clustering or some sort of trend analysis of the transactions. For this I'm seeking advice on what sorts of mechanics I can use - and "gotchas" on what shape the data needs to be. Currently, I guess one "grouping" is that a given set of transactions belong to a given user, and a given user is flagged.

  • Would this be something like PCA or tSNE?
  • How would I stop an algorithm from re-clustering transactions to a given user anyway (because those rows will share the same user_id)? Would I remove the column containing these user IDs?

For the second point, I guess I can use the identified clusters to compared with similar clusters of transactions belong to unflagged users, and see if there are similarities? Would this be something like k-Means?

If you have any questions or require any further information from me, please do let me know.

  • $\begingroup$ This is an interesting problem. Do you know when they were flagged as risky? (i.e. so you could use just the transactions before this point?) I'd start by reversing the question: what are the characteristics of transactions made by risky vs nonrisky users? That way you can duck for a bit the fact that you have a different number of transactions for each user. $\endgroup$ – zbicyclist May 3 at 12:33
  • $\begingroup$ That's a really good question ("when" they were flagged) - I shall ask and see what I receive. Any suggestions on how I can compare characteristics of the transactions between these two groups of users? Initially I was just manually observing the transaction histories of the risky users - because there aren't many, but that won't scale for the many, many nonrisky ones! $\endgroup$ – Chupa May 3 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.