4
$\begingroup$

Trying to decode this task:

I have a table of user ids and the up to 12 products they’ve used over a given period. Two columns, each line being a unique ID and product combo. Boss wants to know how the use of each product relates to each other product, as in: do users of product a also use product c, d but avoid g, h?

That makes me think I need to check correlation, like with a seaborn pairplot, for example.

Is this the/a correct way to proceed? And if so, how does one measure correlation within one field? Would this require an intermediate step of aggregation, since the data I'm analysing isn't table values, but the aggregate of them?

Hopefully this makes sense, tho I'm happy to clarify where needed.

Any thoughts or pointers would be much appreciated. M

$\endgroup$
4
  • $\begingroup$ To clarify, you have 13 columns of data, one with ID and the rest with ether Y or N (or 1 or 0) for the different products? $\endgroup$
    – Peter Flom
    Commented Nov 23, 2023 at 14:49
  • $\begingroup$ Two columns: user_id, product name. There are unique 12 products and about 12k users. I could expand it out to 13 columns if the analysis method required tho. $\endgroup$ Commented Nov 23, 2023 at 14:52
  • $\begingroup$ @MatthewJJJ does each (user, product) tuple always appear 0 or 1 times, or is "how many uses?" a valid question? $\endgroup$
    – hobbs
    Commented Nov 24, 2023 at 4:23
  • $\begingroup$ @hobbs - each user can be paired with up to all 12 products, tho most of them have used 1-3. $\endgroup$ Commented Nov 24, 2023 at 11:44

3 Answers 3

7
$\begingroup$

First, I would expand it to 13 columns.

Then, while you could do some measure of association for each combination of products, that would give you $12\times 11/2 = 66$ measures. Hard to interpret, prone to type 1 error.

I would look into multivariate approaches such as cluster analysis. This thread is a good place to start.

$\endgroup$
2
  • $\begingroup$ Thanks for that Peter - I'll give that a thorough look. Is cluster analysis not more suited for data sets where the values are more variable, though? For example where for the 13 columns, the 12 numerical ones had values other than just 0 or 1? $\endgroup$ Commented Nov 23, 2023 at 15:14
  • $\begingroup$ There are approaches to clustering that work well with dichotmous variables, as in the thread I listed. $\endgroup$
    – Peter Flom
    Commented Nov 23, 2023 at 15:25
3
$\begingroup$

Instead, consider employing association rule mining techniques, such as Apriori or FP-growth algorithms. These methods, commonly used in market basket analysis, can reveal meaningful associations between products. Convert your data into a binary format (used or not used) and use metrics like support, confidence, and lift to interpret the strength of associations.

$\endgroup$
1
  • 1
    $\begingroup$ One option for an existing implementation of frequent pattern mining is mlxtend. $\endgroup$
    – Galen
    Commented Nov 23, 2023 at 16:45
3
$\begingroup$

Mostly out of curiosity, I would explore a random network approach.

Suppose a random graph $G$ with a fixed vertex set $v(G) = \{ \text{products} \}$ representing your set of products and a random edge set $E(G) \subseteq v(G) \times v(G)$. Your random edge set can be represented as a random adjacency matrix $A$ where $A_{ij}$ representing a Bernoulli variable for whether the $i$th product was paired with the $j$th product for some unique ID. Since this would be an undirected graph, you would really only need to have distinct parameters for the upper/lower triangle of this adjacency matrix (and exclude the diagonal since products always co-occur with themselves).

Here is a just a toy specification to get you started, but I mostly will defer development advice to Gelman et el. 2020.

$$\alpha_{ij} \sim \text{Exponential}(1)$$

$$\beta_{ij} \sim \text{Exponential}(1)$$

$$p_{ij} \sim \text{Beta}(\alpha_{ij}, \beta_{ij})$$

$$A_{ij} \sim \text{Bernoulli}(p_{ij})$$

You'll want to expand this first approximation to account for other aspects of the problem. For example:

  • You may want to use a multilevel/mixed effect/hierarchical modelling approach for repeated customers or similarly accounting for a taxonomy on the types of products.
  • There may be further covariance to model between the parameters due to cliques. McElreath 2023 provides some useful guidance on a related problem.
  • There could also be some non-stationarity over time that turns this into a time series problem (if you can get dates) because customers sometimes change their habits.

After sampling from the posterior distribution, you can interpret you $p_{ij}$ as a tendency for the $i$th product to co-occur with the $j$th product. For something analogous to correlation, you could compute the posterior distribution of normalized independence gaps using the sampled values of $p_{ij}$ and marginalization (to obtain the product of the marginals $p_i p_j$).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.