Some media websites have a collection of descriptive tags paired to every image or video, which are intended to help users filter their searches. Such data are observational, so I would limit an analysis to exploring the properties of a (large) sample without inferring causal effects. Rather, I am interested in hypothesis generation from an exploratory data analysis.

For each tag we can create an indicator function $I_{x\in A_i}(x)$ for the $i$th tag. I am interested in learning which tags tend to co-occur, and which tags to tend to not co-occur. Observed groupings of tags might suggest, but not rigorously demonstrate, possible groupings of content preferences.

I could calculate joint frequency probabilities and compare them to the marginals, or perform frequent pattern mining, or one of various clustering algorithms. From a technical standpoint, I essentially have a collection of one-hot vectors. I am wondering what is a suitable approach for finding groupings with such data based on co-occurrence.

  • 2
    $\begingroup$ How many tags are there? Is it a small subset, or a large, growing set like in the case of StackExchange? $\endgroup$
    – Tim
    Commented Sep 29, 2022 at 5:13
  • 2
    $\begingroup$ @Tim Over ten thousand tags across a couple of million observations. I have not checked yet how many are Hapax legomenon, but some quick checks suggest the tags might be power law distributed... Maybe Zipf, specifically, but I have not checked that yet. $\endgroup$ Commented Sep 29, 2022 at 5:17
  • $\begingroup$ You make a logical jump from video tags to personal preferences. Unless you have data about views, or even better, data on views by individual viewers, it's not clear how you'd learn about personal preferences from co-occurrences of labels. It's not even clear whether the labels are assigned by the viewers themselves, so actually the labels might reflect the video platform's guess/knowledge about user preferences. So you might end up doing analysis of their analysis. $\endgroup$
    – dipetkov
    Commented Sep 29, 2022 at 10:37
  • 1
    $\begingroup$ tags sometimes "beg the question". If the customer is running after the keyword, then the association is with the word-symbol, and not necessarily the actual product. I like dark beer but I'm a little picky, so I can look in that section of a supermarket, and find nothing, but if it was an online store they might infer that I connect whatever it was I was looking at before with dark beer, when in fact their dark beer selection was so bad I went elsewhere. User interface design has a huge impact, and the evolution of the amazon webstore UI over time shows that there is still a lot to learn. $\endgroup$ Commented Oct 12, 2022 at 19:48

2 Answers 2

  • You can use the market basket analysis, a simple and popular data mining technique that does exactly what you described: groups co-occurring items together.
  • For a more sophisticated solution, you can use cluster analysis. For binary data, there is a rich family of models called latent class analysis that are designed for this purpose. I'm not sure though if it would scale to a large dataset with many columns and rows like yours.
  • You could transform the binary data into embeddings that would reduce its dimensionality and convert it into continuous features that later could be clustered, which makes it a simpler problem. For creating the embeddings, you could use a neural network or take one of the latent matrices created by matrix factorization (an algorithm used usually for recommender systems), where both scale very well to large data. They will create continuous features, so you can cluster them with any algorithm, e.g. $k$-means.

If you are after visualizing the relationships, then you might use some network visualization algorithms.

I did this once with the tags of cross validated (Disclaimer: I am not an expert at this). I made a matrix that keeps track of the number of times of times that the different tags are appearing together. Then I used that matrix as

  1. a matrix describing the edges in a network graph
  2. a correlation matrix in PCA and
  3. 3 after some arbitrary trial and error with transformations, as a dissimilarity matrix in an hierarchical clustering method

relations between tags


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.