The tag has no usage guidance.

learn more… | top users | synonyms

27
votes
7answers
2k views

Statistical classification of text

I'm a programmer without statistical background, and I'm currently looking at different classification methods for a large number of different documents that I want to classify into pre-defined ...
17
votes
2answers
4k views

Measuring Document Similarity

To cluster (text) documents you need a way of measuring similarity between pairs of documents. Two alternatives are: Compare documents as term vectors using Cosine Similarity - and TF/IDF as the ...
7
votes
2answers
1k views

Can one use Cohen's Kappa for two judgements only?

I am using Cohen's Kappa to calculate the inter-agreement between two judges. It is calculated as: $ \frac{P(A) - P(E)}{1 - P(E)} $ where $P(A)$ is the proportion of agreement and $P(E)$ the ...
6
votes
1answer
118 views

Why does Lucene IDF have a seemingly additional +1?

From the Lucene docs $\text{IDF} = 1 + \log\left(\frac{\text{numDocs}}{\text{docFreq}+1}\right)$ In other references (i.e. wikipedia), IDF is typically calculated as ...
3
votes
2answers
247 views

Dataset and papers for baseline [closed]

I'm doing a project about Topic Detection and Tracking in text. I need to perform a baseline so I can compare existing results with mine. I read some papers where they use datasets that are not so ...
3
votes
0answers
32 views

Heuristics streaming data matching

I have an index composed by thousands of documents. Slightly modified copies of those documents are sent to my application in small chunks, and I need to check, from those chunks, which document has ...
3
votes
2answers
107 views

Vector Space Model for Online News Clustering

I am trying to automatically cluster news articles based on their content. I need this algorithm to be online and simply group news articles related to the same story as they arrive. The common ...