Cluster users based on file access and application usage

Question

I have access to file access data for employees in my organisation. For each employee I can see what read/write operations they carried out in the last month or so. Per operation, I see a user ID, server name, file share, directory and a flag to say if it was a Read or Write operation. I have transformed this data into one instance per user, with one column for the user id and subsequent columns for each server/fileshare/directory combination, with a 0/1 to indicate whether a user has accessed this location in the last month. From another table, I have added additional columns for business applications (there are thousands of apps), also with a 0/1 indicator to indicate whether or not a user uses a particular application.

I would like to cluster these users by similarity based on their systems access and file access profile. The ultimate aim is to see if there are users in, say, South-Africa, who have similar profiles to users in Russia. (the countries here are arbitrary). I've played around with hclust and various distance metrics in R with little success.

This is new to me and I'm just trying to understand what I should be taking into account when identifying these clusters. Are there any particular clustering techniques and distance metrics that would or would not work well for this type of problem ? Instead of using the 0/1 indicator, should I be using the total number of access counts per user per location ?

Answer 1

I agree that some sort of dimension reduction approach of your features makes sense. Standard PCA was developed for finding linear combinations of continuously scaled features based on symmetric matrices and wouldn't be appropriate for your many 0,1 features. When faced with matrices of categorical data that are not symmetric, sparsely populated or potentially nonlinear, SVD-based factorization methods are a much better, more robust approach. Correspondence analysis is one SVD method for finding low dimensional, "PCA-like" solutions based on categorical features but there are many others. One recent paper by Chen and Xie uses a probabilistic, robust Cauchy model rooted in SVD decomposition designed to capture nonlinear, sparse and extreme valued information...

http://arxiv.org/abs/1412.6506

Bayesian Tensor Regression, an approach developed by David Dunson at Duke is kind of the state-of-the-art for massive, multiway contingency table analysis. But it's not clear that this would be a useful approach for your needs since, if the prior responder is right that you have "many irrelevant features", it takes the dimensionality as fixed as in genome analysis. You should be aware that the approach exists.

http://www.researchgate.net/profile/David_Dunson/publications

Once you have reduced the dimensionality and scored your employees, then there are many clustering algorithms available for use. K-means is the granddaddy of them all but has the limitation of always finding spherically-shaped clusters, even when there really aren't any real groupings to the data. But it works pretty well. K-medians would be another option. Agglomerative methods will find differently shaped cluster segments and can be based on similarities between the dimensions. But there are almost as many clustering algorithms as there are variable selection algorithms -- both are areas where every statistician and their brother has an approach and/or a working paper.

Answer 2

You may have too many irrelevant attributes. Due to the curse of dimensionality the similarities do not work well anymore.

Some methods are more affected than others. In particular approaches that look at all attributes and all objects are prone to such problems.

You could try Apriori to find frequent combinations instead.