The tag has no wiki summary.

learn more… | top users | synonyms

5
votes
1answer
309 views

Choosing a measure of similarity to quantify similarity between individuals on a set of personality scales

I have a bunch of users. Each user has a number of personality attributes, such as "fitness level" or "eco-consciousness", rated on a scale from 1 to 5. I want to calculate how similar two users are, ...
2
votes
0answers
154 views

How to to calculate the topic distribution of a document

I have a simple (may be stupid) question. I want to calculate KullbackÔÇôLeibler divergence on two documents. It requires probability distribution of each document. I do not know how to calculate ...
3
votes
1answer
658 views

Using latent Dirichlet allocation for information retrieval

I am working on understanding various document ranking algorithms like (TF-IDF, LSI, language models, etc) by actually implementing them. I want to understand LDA and using various resources to ...
4
votes
2answers
419 views

How to compute term frequency and find clusters in a dataset composed of strings?

I am currently looking for some Information Retrieval techniques. I have a SQL database table containing strings. It has 1000 records, each being a random sentence I picked from random web sites. I ...
4
votes
2answers
1k views

Comparing cosine similarities for tf-idf vectors for documents with different length

I'm computing cosine similarities between 2 vectors. These vectors are information retrieval query and document representations respectively. They have been computed using tf-idf weights. Since my ...
2
votes
2answers
563 views

KL divergence calculation

I am wondering that how one can calculate KL-divergence on two probability distributions. For example, if we have ...
7
votes
2answers
1k views

Can one use Cohen's Kappa for two judgements only?

I am using Cohen's Kappa to calculate the inter-agreement between two judges. It is calculated as: $ \frac{P(A) - P(E)}{1 - P(E)} $ where $P(A)$ is the proportion of agreement and $P(E)$ the ...
13
votes
1answer
3k views

Measuring Document Similarity

To cluster (text) documents you need a way of measuring similarity between pairs of documents. Two alternatives are: Compare documents as term vectors using Cosine Similarity - and TF/IDF as the ...
23
votes
7answers
1k views

Statistical classification of text

I'm a programmer without statistical background, and I'm currently looking at different classification methods for a large number of different documents that I want to classify into pre-defined ...