Cluster Groups based on overlap I have the following data:
Group_id  item_id
     001      699
     001      533
     001      782

     002      699
     002      533
     002      782
     002      999

I want to assign group_id 001 and 002 in one cluster because they have a large overlap between item_id's.
Group_id and item_id are just arbitrary numbers. So I am not sure if it makes sense to calculate distance between the item_ids numbers?
 A: Since you mentioned "hierarchical-clustering" in the tags I assume you want to use hierarchical clustering for this task.
Distance Matrix
Hierarchical clustering can cluster groups based on their pairwise distance matrix. So what you need to do is to prepare a matrix of distances between all pairs of group_ids.
Distance between each pair
In order to construct the distance matrix you need some way to measure the distances between each pair of group_ids. In your scenario you want to cluster groups based on the overlaps of their elements. One way to do this would be the Jaccard similarity which is an intersect over a union of elements.
With your example the Jaccard similarity between group_ids 001 and 002 would be computed like so:


*

*Find the number of common elements: in your case - 3

*Find the number of total unique elements in both groups: in your case - 4

*Jaccard's similarity is 3/4


Note that Jaccard's similarity shows how similar two groups are. You will probably need to turn this into a matrix of distances. Since Jaccard's similarity ranges from 0 to 1 - you can transform them into distances by subtracting them from 1: 1 - Jaccard similarity
Hierarchical Clustering
Once you have the required distance matrix - you can continue with hierarchical clustering. There might be some parameters left to decide on (like a linkage function to use when combining two groups into a single cluster) but these are mostly problem dependant.
