The information-retrieval tag has no wiki summary.
0
votes
0answers
25 views
Unusual Precision-Recall Curve
Looking at precision-recall curves to evaluate an information retrieval system. Typically, these are done with a set of samples, with what users have picked out of a set of given items.
I have an ...
0
votes
0answers
30 views
Inferring user rating from play counts
I am interested in converting user play counts in some sort of rating. Building an artist recommendation system and I have access to user play counts. I can't use them directly in a user-based ...
1
vote
0answers
33 views
MAE/MSE with or without square root
I read some papers about recommender systems and information retrieval, where Mean Absolut Error and Mean Squared Error are mentioned. But I've found some differences between the formal definition of ...
0
votes
1answer
22 views
Discounted cumulated gain
I've a little question regarding the Discounted Cumulated Gain (DCG) (Sorry, I couldn't find the papers of Järvelin and Kekäläinen). Can this evaluation-metric be used when a information retrieval ...
4
votes
2answers
125 views
Summary statistics of the precision-recall curve
From what I understand, one can use the AUC of the ROC curve as a summary statistic of the full curve.
Q1. Are there any similar summary statistics that one can use on a single precision-recall ...
1
vote
0answers
43 views
Is that overfitting?
Given a document-term matrix $X$, where $$X(d, t) = \textit{occurrences of 't' in 'd'}$$, it's possible to compute it's Truncated Singular Value Decomposition:$$X_k = U_k \Sigma_k V_k^T$$
Then, for a ...
0
votes
3answers
86 views
Is there any dataset or api that gives a list of infrequent words? [closed]
I'm actually working on an information retrieval project, and I want to extract words that are of significance from a sentence. This is somewhat opposite to stopwords.
In a sentence like: He was a ...
2
votes
1answer
204 views
A parellel between LSA and pLSA
In the original paper of pLSA the author, Thomas Hoffman, draw a parallel between pLSA and LSA data structures that I would like to discuss with you.
Background:
Taking inspiration the Information ...
1
vote
0answers
51 views
Using sentiment lexicons or all words processing for sentiment analysis?
I am learning sentiment analysis to apply it to twitter real time data to predict user's mood. I ponder about using which alternative way to do that data mining job.
Use all words to process and ...
1
vote
0answers
24 views
Evaluation and Testsets for NNMF
I am trying to evaluate my recommender system which uses Non-negative Matrix Factorization. Some things that I evaluate are
How does the size of the feature matrix affect the recommendations
How ...
3
votes
1answer
668 views
Euclidean Distance b/t unit vectors or cosine similarity where vectors are document vectors
I was reading Similarity Measures and suddenly my whole world was falling apart. I have implemented a search engine using clustering techniques. For clustering, I used k means which uses Euclidean ...
2
votes
2answers
116 views
How do you predict a continuous value from many booleans & a continuous value?
Hello: I am a computer science student working as a research assistant in an undergrad IR lab, feeling spectacularly out of my element.
Given an input of a single continuous value and a vector of ...
2
votes
0answers
62 views
Software library for Hidden Markov Modeling of a large text database
Given
we have a large database of texts (e.g. product descriptions)
and we want to extract multiple types of information (e.g. brand, release date, features, price, etc.)
what's a good library to ...
7
votes
2answers
206 views
Understanding and applying sentiment analysis
I was just having been assigned a project of conducting sentiment analysis for some document collections. By Googling, a lot of sentiment-related research has popped up.
My questions are:
What are ...
2
votes
0answers
238 views
Similarity calculations for arrays
First of all, my apologies if I mess up the terminology. I've been out of math for several years, so I'm certain I'm going to use terms incorrectly. Also, though I concentrated mathematics in college, ...
5
votes
1answer
203 views
Choosing a measure of similarity to quantify similarity between individuals on a set of personality scales
I have a bunch of users. Each user has a number of personality attributes, such as "fitness level" or "eco-consciousness", rated on a scale from 1 to 5. I want to calculate how similar two users are, ...
2
votes
0answers
129 views
How to to calculate the topic distribution of a document
I have a simple (may be stupid) question. I want to calculate Kullback–Leibler divergence on two documents. It requires probability distribution of each document.
I do not know how to calculate ...
3
votes
1answer
492 views
Using latent Dirichlet allocation for information retrieval
I am working on understanding various document ranking algorithms like (TF-IDF, LSI, language models, etc) by actually implementing them. I want to understand LDA and using various resources to ...
4
votes
2answers
312 views
How to compute term frequency and find clusters in a dataset composed of strings?
I am currently looking for some Information Retrieval techniques.
I have a SQL database table containing strings. It has 1000 records, each being a random sentence I picked from random web sites. I ...
3
votes
2answers
695 views
Comparing cosine similarities for tf-idf vectors for documents with different length
I'm computing cosine similarities between 2 vectors.
These vectors are information retrieval query and document representations respectively.
They have been computed using tf-idf weights.
Since my ...
1
vote
2answers
403 views
KL divergence calculation
I am wondering that how one can calculate KL-divergence on two probability distributions. For example, if we have
...
7
votes
2answers
840 views
Can one use Cohen's Kappa for two judgements only?
I am using Cohen's Kappa to calculate the inter-agreement between two judges.
It is calculated as:
$ \frac{P(A) - P(E)}{1 - P(E)} $
where $P(A)$ is the proportion of agreement and $P(E)$ the ...
9
votes
1answer
2k views
Measuring Document Similarity
To cluster (text) documents you need a way of measuring similarity between pairs of documents.
Two alternatives are:
Compare documents as term vectors using Cosine Similarity - and TF/IDF as the ...
22
votes
6answers
886 views
Statistical classification of text
I'm a programmer without statistical background, and I'm currently looking at different classification methods for a large number of different documents that I want to classify into pre-defined ...
