I have a large collection of documents, and would like to be able to select a subset of them that is representative of the whole. I have searched this question on here, Stack Overflow, and Google, without getting an answer I liked. (In particular, most of the answers I've seen deal in topics, and I'm not a huge fan of topic modeling.) I have a heuristic procedure in mind, and was curious if it seemed sensible -- or, if the idea inspired a better method. My question is most similar to this one, Can I apply word2vec to find document similarity? but with a few more layers, and a bit more statistics.

In my own understanding, the core statistical issue is How do I compare (measure distances between) moderately-sized, high-dimensional clusters of points. However I include the context in case you see a way around it that avoids this problem entirely.

As is, I'm sure, often the case, much of the question has to do with the appropriate representation of a document. While not being very familiar with its properties, I greatly prefer the intuitiveness of word2vec.

The Proposed Procedure

1) Train a word2vec model using the pooled corpus (and possibly, add in another larger hopefully similar corpus, if the one I have is too small).

2) Replace every (non "stop") word in each document with its word vector representation, producing a single N x K matrix (where N is the number of words, and K is the dimension of the vectors).

3) Create an additional set of columns by multiplying all of the existing columns together in pairs. Each document matrix is now approximately N x (K+K^2). This creates interaction terms for the latent dimensions, explicitly encoding information from the embedding that will be destroyed otherwise in subsequent steps.

4) Because documents of different lengths will be represented by matrices of different sizes, and the goal is to compare them, something must be done to make the matrices comparable. I propose making a histogram of each column. Say each histogram is made with B bins, then the previously N x (K+K^2) will become B x (K+K^2). This will destroy associations between columns, which is why I made the interaction columns, above.

5) Now that everything is an equally sized matrix, I can easily measure distances with, for ex., the Frobenius norm, give the distances to a clustering algorithm, and select the centroids of each cluster as the representative subset.

There are more straight-forward treatments of the original document matrices, but I don't know how to compare them. I could throw them, for example, into kPCA, but I don't understand it well enough to know if the results could be compared meaningfully. Could I meaningfully compare the projections that came from two different embeddings?

  • $\begingroup$ You introduced a new tag [kpca]. Can you please write a tag wiki? $\endgroup$ – kjetil b halvorsen May 7 '18 at 9:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.