I have a high level question about whether or not graph networks would be an appropriate method to model a situation I'm studying. It's been a while since I last worked on a project building/analyzing a network, so I just wanted to know if this is worth continuing to explore and learn about.
So far, I've been looking into sales data and have been clustering products that are frequently purchased together. I'm more familiar with traditional unsupervised clustering algorithms and frequent item set mining. Some of the future goals of this project are to identify broad clusters of products that are frequently purchased for the same application and to provide product recommendations.
Here's what I'm working with:
- I have data for 10,000s of purchase orders. Most purchase orders contain 3-10 product SKUs which were purchased together at one time.
- There are 10,000s of product SKUs, but the vast majority of these (all except a few thousand) are very rarely purchased and can be probably be excluded for now.
- For the vast majority of products, I don't know their categorization, but I can manually look them up and figure out what they are used for.
- Like many product sets, there are a small handful of products that are purchased very frequently, and a long tail of products that are rarely purchased.
- The most frequently purchased products (top 100 or so) generally have 2 major applications. Some of the top products are specifically for "Application A" while other products are for "Application B". It is rare that someone would purchase items for both "Application A" and "Application B" at the same time, but it does occasionally happen. Some products (esp those in the long tail) can be used for either application.
- One of the ways I've been evaluating my clustering results was to see if "Application A" and "Application B" products were generally separating into different clusters.
I was using Python's networkx and R's igraph to model the products as nodes and the weighted edges as the count of times those products were purchased together (or a variation of that value).
On a smaller scale with a few dozen invoices, the graph seemed to work decently. I could visually see a few communities of complementary products. However, I'm having more difficulty when using the entire data set. Just about every product has been purchased with every other product at least once. I end up with one massive cluster of products and a few other very small clusters. I've played around with the weight calculation & similarity measurements and removed edges below a certain threshold, but I still haven't gotten useful insights.
Has anyone worked on a similar problem? How did you weight the edges? Is this a solution I should keep exploring, or would the algorithms found in scikit and frequent item set mining be more appropriate? Thanks.