The tag has no usage guidance.

learn more… | top users | synonyms

29
votes
7answers
2k views

Statistical classification of text

I'm a programmer without statistical background, and I'm currently looking at different classification methods for a large number of different documents that I want to classify into pre-defined ...
17
votes
2answers
6k views

Measuring Document Similarity

To cluster (text) documents you need a way of measuring similarity between pairs of documents. Two alternatives are: Compare documents as term vectors using Cosine Similarity - and TF/IDF as the ...
7
votes
2answers
2k views

Can one use Cohen's Kappa for two judgements only?

I am using Cohen's Kappa to calculate the inter-agreement between two judges. It is calculated as: $ \frac{P(A) - P(E)}{1 - P(E)} $ where $P(A)$ is the proportion of agreement and $P(E)$ the ...
6
votes
1answer
171 views

Why does Lucene IDF have a seemingly additional +1?

From the Lucene docs $\text{IDF} = 1 + \log\left(\frac{\text{numDocs}}{\text{docFreq}+1}\right)$ In other references (i.e. wikipedia), IDF is typically calculated as $\log\left(\frac{\text{numDocs}}{...
3
votes
2answers
405 views

Dataset and papers for baseline [closed]

I'm doing a project about Topic Detection and Tracking in text. I need to perform a baseline so I can compare existing results with mine. I read some papers where they use datasets that are not so ...
3
votes
2answers
188 views

Vector Space Model for Online News Clustering

I am trying to automatically cluster news articles based on their content. I need this algorithm to be online and simply group news articles related to the same story as they arrive. The common ...