Identifying a single cluster from multiple clusters per customer

Question

Although I have worked out several possible approaches, am curious if anyone has a better best practice to recommend?

I have a large set of customer events (online behavior: 1 mil. records, 20 standardized continuous variables, 8000 customers). I am seeking to identify unique clusters/groups of customers based upon many events per customer.

I have derived solid clusters from the events (good R2 and low misclassification in log regression hold out testing). However, this leads to many customers being in multiple clusters. I want to ultimately collapse the event clusters into customer unique clusters.

One approach is to average per customer and to cluster on averages (or z-score) from the start. Not happy with this due to information loss. For instance, some customers have many events and some just a few.

Another approach I attempted was to cluster the whole dataset such that some customers were identified with multiple clusters and then to assess matrix covariance in cluster membership between customers to derive a new set of groups using graph community detection (modularity).

This was not ideal as there is information loss and also have not seen reflected in the literature - I made up the approach and am convinced there must be a better way.

Essentially it is the issue of clustering many events tied to many customers (without regard to customers initially) to seek similar behaviors, then attempting to reduce clusters so each customer is fit to one cluster.

It appears to be a two-stage clustering problem: cluster on events, then cluster on some aspect of the customers in order to produce customer clusters which trap variability in customers (particularly those with many events and/or highly diverse behavior).

One approach would be to average cluster averages per customer (roll-up multiple cluster averages per customer then cluster on averaged measures per customer). However, this dispenses with number of events and standard deviation.

It seems to me it might be a fairly common problem that those with more experience in marketing analytics may have run across and handled in a best practices way. Have not yet found any guidance, but perhaps not looking in right places or framing question clearly.

Greatly appreciate any thoughts / wisdom on this topic!

Answer 1

This doesn't constitute an answer, just a thought. Perhaps there are two outcomes you need to infer - customer cluster and say purpose for visiting (if web events for example). The combination of 'customer cluster' and 'purpose' leads to 'events' which you have already defined by your event clusters. Maybe this can be solved if you fix the number of customer clusters and minimize the number of 'purposes' such that there is a one-to-one relationship from cluster and purpose to event.

As you say though, this seems like a common problem so there are probably 'proven' ways of doing this.