0
$\begingroup$

A modest attempt to illustrate my question:

Setup

Consider a survey where you have to choose if you like to do an activity or not. You do this for a number of activities $A_1,...,A_8$. In addition, we know a range of possible jobs (let us say two, to keep it simple), which are associated with a number of activities. We ask three people $(P_1,P_2,P_3)$ to tick the activities they like and we assign the activity a $1$ if chosen and a $0$ if not. We can then list the recorded answers along with the potential jobs, denoted as $J_1$ and $J_2$.

\begin{array}{llll} \text{Activity} & P_1 & P_2 & P_3 && J_1 & J_2\\ \hline A_1 & 1 & 1 & 1 && 1 & 0 \\ A_2 & 1 & 1 & 1 && 1 & 0 \\ A_3 & 1 & 1 & 0 && 1 & 0 \\ A_4 & 1 & 0 & 0 && 0 & 1 \\ A_5 & 1 & 0 & 0 && 0 & 1 \\ A_6 & 1 & 0 & 1 && 0 & 1 \\ A_7 & 1 & 0 & 1 && 0 & 1 \\ A_8 & 1 & 0 & 1 && 0 & 1 \end{array}

Considerations

We want to know what job suits each person best based on their choice of activities. One way to do so is to calculate the simple Euclidean distance between all possible pairs of $P$ and $J$ and determine the closest $J$ for each $P$ in the 8-dimensional space (a k-nearest neighbor approach). We then get:

\begin{array}{lll} & J_1 & J_2 \\ P_1 & \sqrt{5} & \sqrt{3} \\ P_2 & 0 & \sqrt{8} \\ P_3 & 2 & 2 \end{array}

We realise that it is impossible to decide on the best job for person 3 since $d(P_3,J_1)=d(P_3,J_2)$. However, we decide to only consider activities relevant to the job when calculating the distance, and re-calculate $d(P_3,J_1)$ for only the first three activities and $d(P_3,J_2)$ for the last five activites. This gives us $d(P_3,J_1)=1$ and $d(P_3,J_2)=\sqrt{2}$, hence preferring $J_1$ over $J_2$ for $P_3$.

Question

In the above example, the nearest neighbor for $P_3$ is $J_1$ if we calculate the distance only for relevant activities. However, the number of dimensions vary, and a direct comparison of the two distances does not seem valid. Is there an obvious method/workaround here that I have missed? When looking at previous questions about calculating the distance between vectors of different dimensions, some mentions projecting the lower-dimension vector onto the higher-dimension vector by injecting 0's, which is not really an option here. The original problem consists of around 150 "activities" on a scale from 0-3 and 500 "jobs". Any suggestions are much appreciated.

$\endgroup$
0
$\begingroup$

Jaccard similarity might be better suited for your use case.

https://en.wikipedia.org/wiki/Jaccard_index

$\endgroup$
  • $\begingroup$ Would you care to elaborate? As far as I can tell, we are still left with the choice of calculating two different similarities. One for the whole set, and one for the set which is relevant only for a given job. $\endgroup$ – thesixmax Jan 11 at 12:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.