Distance calculation on variables that cannot be represented in the Euclidean space

Question

Given a variable such as number of events attended together, which is more of a multi-dimensional data how can you calculate a sort of distance between people (i.e. a similarity score)?

Context:

For simplicity, lets say there are two types of data available

1. Customer_id and their purchases across brands (a.k.a. Buying Patterns)
2. Customer_id and the events attended (a.k.a. Socializing Patterns), where a customer may attend multiple events

The idea is to process the data and get a weighted (Based on business priorities) average similarity score that can identify customers who are similar to each other.

1. Buying Patterns: Here because it is continuous data that can be rolled up at customer_id level we can simply take a distance measure.
2. Socializing Patterns: If we roll up the data to a customer_id level with the number of events attended we lose information about which customer attended the same event as the other. This is valuable info lost as they might have met each other, may have mutual friends attending the event or simply means that they have similar tastes.

I was thinking of simply taking the number of events attended together itself as a distance between them after reversing the values (i.e. Maximum value - value), assuming that people who have attended more number of events together are similar to each other.

Is there a better approach to this? A better way to handle a variable that is more of a network variable (if that is the right word)

Note: there are 100s of events, and when I mean the distance cannot be represented in the Euclidean space I mean you cannot simply calculate the distance on the events data.

For example given an input with customer_id, event_id how can you measure the similarities between customers? You can't simply count the number of events attended and then calculate the distance because of the problem I've mentioned above.

One idea is to try out Gower's measure but I'm still trying to understand what that does exactly.

Answer 1

I assume you store your customers in any form similar to this:

(customer_1, {event_1, event_5, event_7, event_8})
(customer_2, {event_2, event_3, event_5, event_8, event_13})
...

Then you can measure how similar a given pair of customers are to each other, by measuring a similarity score for the respective sets of attended events.

Measuring similarity of sets with a notion of overlap can be done with Jaccard coefficient or Dice coefficient (among others). These are related, and can be derived from each other. You should check which one works best for your data, empirically.

Let A, B be the sets of events attended by customer 1, and customer 2, respectively.

Jaccard measures the number of commonly attended events, normalized by the number of events attended by either person: $$ J(A,B) = {{|A \cap B|}\over{|A \cup B|}} = {{|A \cap B|}\over{|A| + |B| - |A \cap B|}} $$

Dice measures the number of commonly attended events (times 2), normalized by the number of events attended by either person with repetition: $$ D(A,B) = \frac{2 |A \cap B|}{|A|+ |B|} $$

In the above example, the coefficients are: $J(A,B)=2/7$ and $D(A,B)=4/9$

Implementations are widely available (e.g., scikitlearn, Java Apache commons for Jaccard), but can also be done very easily on your own for any general programming language.

Answer 2

I would suggest using the sample Mahalanobis distance for the vectors of indicator variables for the events. Suppose that you have $n$ customers and $m$ social events, so that you can represent event attendance in an $n \times m$ matrix:

$$\mathbf{M} = \begin{bmatrix} \boldsymbol{m}_1^\text{T} \\ \boldsymbol{m}_2^\text{T} \\ \vdots \\ \boldsymbol{m}_n^\text{T} \end{bmatrix} = \begin{bmatrix} m_{1,1} & m_{1,2} & \cdots & m_{1,m} \\ m_{2,1} & m_{2,2} & \cdots & m_{2,m} \\ \vdots & \vdots & \ddots & \vdots \\ m_{n,1} & m_{n,2} & \cdots & m_{n,m} \\ \end{bmatrix}.$$

The elements of this matrix would be indicator variables, but this is not actually necessary for the proceeding calculation of the distances. The matrix of squared Mahalanobis distances between the customeres is an $n \times n$ matrix given by:

$$\mathbf{D}^2 = \frac{\mathbf{c} \mathbf{M} (\mathbf{M}^\text{T} \mathbf{c} \mathbf{M})^{-1} \mathbf{M}^\text{T} \mathbf{c}}{n-1} \quad \quad \quad \mathbf{c} = \mathbf{I}_n - \tfrac{1}{n} \mathbf{1}_{n \times n}.$$

(Note that the matrix $\mathbf{c}$ is the centering matrix.) The square roots of the values in this matrix give the distances between the customers in terms of their event attendance.

Implementation in R: The above squared-distance matrix $\mathbf{D}^2$ can be calculated in R using the mahalanobis function in the stats package. This is done as follows (this code assumes that you have already entered the data as an $n \times m$ matrix of indicators assigned as M):

library(stats);
D2 <- mahalanobis(M, colMeans(M), cov(M));