2
$\begingroup$

I collected a corpus of $n$ tweets (few thousand) during a 48-period. The tweets were all collected based on a set of search terms. The tweets were published by $a$ authors, with $a \leq n$. Let's define the collection of all tweets published by the same author as a document, the smallest unit in my corpus.

Now I want to understand if it is possible (and makes sense) to compute the distance from each document and the entire corpus. I know that the cosine similarity of term frequencies it is usually applied between documents pertaining to the same corpus (eventually weighted to account for rare words). But how to proceed if I am interested in the 'originality' of each document compared to the other fellow documents of the corpus?

Improve this question
$\endgroup$
10
  • 1
    $\begingroup$ Don't think that the use of document here is appropriate -- each tweet is still a document. If you want to measure an average divergence in author styles, why not compute pairwise distances between each authors' documents, and scale by within author divergence? $\endgroup$
    – tchakravarty
    Mar 26, 2015 at 4:06
  • $\begingroup$ @TC that just answers a different question $\endgroup$
    – shadowtalker
    Mar 26, 2015 at 4:19
  • $\begingroup$ Also this could be a good use case for KL divergence. $\endgroup$
    – shadowtalker
    Mar 26, 2015 at 4:23
  • $\begingroup$ @ssdecontrol You might want to elaborate on that? If you are referring to the deviation from the entire corpus bit, it is the same except now scale by all pairwise divergences. $\endgroup$
    – tchakravarty
    Mar 26, 2015 at 4:24
  • $\begingroup$ @TC it seems the goal is to find "atypical" authors. Computing pairwise author distances doesn't make sense to me here $\endgroup$
    – shadowtalker
    Mar 26, 2015 at 4:26

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.