I collected a corpus of $n$ tweets (few thousand) during a 48-period. The tweets were all collected based on a set of search terms. The tweets were published by $a$ authors, with $a \leq n$. Let's define the collection of all tweets published by the same author as a document, the smallest unit in my corpus.

Now I want to understand if it is possible (and makes sense) to compute the distance from each document and the entire corpus. I know that the cosine similarity of term frequencies it is usually applied between documents pertaining to the same corpus (eventually weighted to account for rare words). But how to proceed if I am interested in the 'originality' of each document compared to the other fellow documents of the corpus?

  • 1
    $\begingroup$ Don't think that the use of document here is appropriate -- each tweet is still a document. If you want to measure an average divergence in author styles, why not compute pairwise distances between each authors' documents, and scale by within author divergence? $\endgroup$ – tchakravarty Mar 26 '15 at 4:06
  • $\begingroup$ @TC that just answers a different question $\endgroup$ – shadowtalker Mar 26 '15 at 4:19
  • $\begingroup$ Also this could be a good use case for KL divergence. $\endgroup$ – shadowtalker Mar 26 '15 at 4:23
  • $\begingroup$ @ssdecontrol You might want to elaborate on that? If you are referring to the deviation from the entire corpus bit, it is the same except now scale by all pairwise divergences. $\endgroup$ – tchakravarty Mar 26 '15 at 4:24
  • $\begingroup$ @TC it seems the goal is to find "atypical" authors. Computing pairwise author distances doesn't make sense to me here $\endgroup$ – shadowtalker Mar 26 '15 at 4:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.