Questions tagged [information-retrieval]

The tag has no usage guidance.

32
votes
7answers
2k views

Statistical classification of text

I'm a programmer without statistical background, and I'm currently looking at different classification methods for a large number of different documents that I want to classify into pre-defined ...
21
votes
2answers
9k views

Measuring Document Similarity

To cluster (text) documents you need a way of measuring similarity between pairs of documents. Two alternatives are: Compare documents as term vectors using Cosine Similarity - and TF/IDF as the ...
9
votes
1answer
10k views

Mean Average Precision vs Mean Reciprocal Rank

I am trying to understand when it is appropriate to use the MAP and when MRR should be used. I found this presentation that states that MRR is best utilised when the number of relevant results is less ...
9
votes
1answer
2k views

A parellel between LSA and pLSA

In the original paper of pLSA the author, Thomas Hoffman, draw a parallel between pLSA and LSA data structures that I would like to discuss with you. Background: Taking inspiration the Information ...
8
votes
1answer
447 views

Why does Lucene IDF have a seemingly additional +1?

From the Lucene docs $\text{IDF} = 1 + \log\left(\frac{\text{numDocs}}{\text{docFreq}+1}\right)$ In other references (i.e. wikipedia), IDF is typically calculated as $\log\left(\frac{\text{numDocs}}{...
8
votes
2answers
685 views

Understanding and applying sentiment analysis

I was just having been assigned a project of conducting sentiment analysis for some document collections. By Googling, a lot of sentiment-related research has popped up. My questions are: What are ...
7
votes
2answers
2k views

Can one use Cohen's Kappa for two judgements only?

I am using Cohen's Kappa to calculate the inter-agreement between two judges. It is calculated as: $ \frac{P(A) - P(E)}{1 - P(E)} $ where $P(A)$ is the proportion of agreement and $P(E)$ the ...
7
votes
4answers
6k views

What is the best algorithm to find similar text documents?

I have many text documents and I would like to find similar documents to each document within my data set. Is Latent Dirichlet Allocation (LDA) the best way to do that, or are there other algorithms ...
6
votes
3answers
1k views

Summary statistics of the precision-recall curve

From what I understand, one can use the AUC of the ROC curve as a summary statistic of the full curve. Q1. Are there any similar summary statistics that one can use on a single precision-recall ...
6
votes
1answer
2k views

The Effect of Stopword Filtering prior to Word Embedding Training

Recently I have played with the pretrained GLOVE word embedding model for Twitter http://nlp.stanford.edu/projects/glove/ I notice that common stopwords are existing in the model. That is, there is ...
6
votes
2answers
4k views

Is it ok to get negative Cosine Similarity using LSA?

I am getting negative cosine similarity value between two documents in Latent Semantic analysis. How should it be treated?
5
votes
2answers
6k views

How do you predict a continuous value from many booleans & a continuous value?

Hello: I am a computer science student working as a research assistant in an undergrad IR lab, feeling spectacularly out of my element. Given an input of a single continuous value and a vector of ...
5
votes
1answer
3k views

Difference between Log Entropy Model and TF-IDF Model?

I would like to understand what are the differences/advantages in using TF-IDF or the Log Entropy model for represeting documents and queries in an information retrieval system using diferent weights. ...
5
votes
1answer
314 views

In inverse theory, how do I transform the averaging kernel matrix to a new grid?

Rodgers and Connor (2003) describe how measurements by remote sounders can be properly compared, taking into account differences in averaging kernels and error covariances. They make the assumption ...
5
votes
2answers
1k views

How to compute term frequency and find clusters in a dataset composed of strings?

I am currently looking for some Information Retrieval techniques. I have a SQL database table containing strings. It has 1000 records, each being a random sentence I picked from random web sites. I ...
5
votes
1answer
402 views

What is the standard procedure for evaluating a user-based CF algorithm with a dataset offline?

I have read some papers and other materials about the evaluation of recommender systems (RS). Most of them discuss the various properties of RS (e.g. accuracy, diversity, etc.), and metrics for ...
5
votes
1answer
1k views

Choosing a measure of similarity to quantify similarity between individuals on a set of personality scales

I have a bunch of users. Each user has a number of personality attributes, such as "fitness level" or "eco-consciousness", rated on a scale from 1 to 5. I want to calculate how similar two users are, ...
4
votes
1answer
851 views

Difference between Weighted Average Entropy and Adjusted Mutual Information (for evaluating Clustering)

I was advised by my team leader to use this weighted average entropy to evaluating the performance of my dbscan clustering algorithm, and its mathematical formulation is: Scikit provides what many ...
4
votes
2answers
4k views

Comparing cosine similarities for tf-idf vectors for documents with different length

I'm computing cosine similarities between 2 vectors. These vectors are information retrieval query and document representations respectively. They have been computed using tf-idf weights. Since my ...
4
votes
1answer
872 views

Normalised score for BM25

BM25 provides a function that assigns a score that is a function of a query and a document. The score is computed for each document in the collection and can be used to rank documents, but this score ...
4
votes
0answers
822 views

Learning to Rank: query-dependent vs. query-independent features

I've been doing some reading about learning to rank - specifically lambdaMART - and one thing I am confused about is the role of features. When training a model, should one only use query-dependent ...
3
votes
1answer
1k views

Using latent Dirichlet allocation for information retrieval

I am working on understanding various document ranking algorithms like (TF-IDF, LSI, language models, etc) by actually implementing them. I want to understand LDA and using various resources to ...
3
votes
2answers
72 views

Is there a ranking metric based on percentages that favors larger magnitudes?

I have two groups, "in" and "out," and item categories that can be split up among the groups. For example, I can have item category A that is 99% "in" and 1% "out," and item B that is 98% "in" and 2% "...
3
votes
1answer
5k views

Euclidean Distance b/t unit vectors or cosine similarity where vectors are document vectors

I was reading Similarity Measures and suddenly my whole world was falling apart. I have implemented a search engine using clustering techniques. For clustering, I used k means which uses Euclidean ...
3
votes
2answers
771 views

Most important journals in data mining/ML, NLP and IR?

Can you please provide with me with the names of the most important journals in data mining, machine learning, natural language processing and information retrieval?
3
votes
1answer
2k views

Both validation loss and accuracy goes up in neural network

I'm training a 2-layer CNN model on audio samples, represented as CQT. There are ≈160k samples, many that are very similar since they originate from the same instrument and/or audio file. 10% have ...
3
votes
2answers
819 views

Dataset and papers for baseline [closed]

I'm doing a project about Topic Detection and Tracking in text. I need to perform a baseline so I can compare existing results with mine. I read some papers where they use datasets that are not so ...
3
votes
2answers
451 views

Vector Space Model for Online News Clustering

I am trying to automatically cluster news articles based on their content. I need this algorithm to be online and simply group news articles related to the same story as they arrive. The common ...
3
votes
1answer
6k views

“Mean average precision” (MAP) evaluation statistic - understanding good/bad/chance values

I'm evaluating a multilabel classifier. I'm familiar with the Area Under the Curve statistic, which has some nice properties (e.g. chance level is always 50%). But for some applications, it's more ...
3
votes
2answers
686 views

Other documents features than tf-idf for clustering?

What are other feature representations for documents that are used for clustering textual documents? The only representation I'm aware of is tf-idf. Are there other ones?
3
votes
0answers
71 views

How can I deal with the mismatch between the vocabularies of questions and answers in a closed domain QA system?

I am building a question answering system that given a legal document attempts to answer questions related to the document. For example a tenancy agreement is given to the system and the user asks ...
3
votes
0answers
101 views

Heuristics streaming data matching

I have an index composed by thousands of documents. Slightly modified copies of those documents are sent to my application in small chunks, and I need to check, from those chunks, which document has ...
3
votes
0answers
74 views

Evaluation of a semi-supervised ranking model

I learned ranking model with graph-based semi-supervised approach, while labeled (just positive) and unlabeled (positive and negative) data is both used in training. With the model, all of the data ...
3
votes
0answers
1k views

Why are ERR (Expected Reciprocal Ranking) scores not normalized?

It seems to me that normalized ERR (Expected Reciprocal Ranking) scores (ERR scores of your ranking algorithm divided by ERR score calculated for the ground truth ranking) are more useful than the ...
3
votes
0answers
430 views

Similarity calculations for arrays

First of all, my apologies if I mess up the terminology. I've been out of math for several years, so I'm certain I'm going to use terms incorrectly. Also, though I concentrated mathematics in college, ...
2
votes
1answer
1k views

Remove bias in ranking evaluation

In ranking, a commonly used method to evaluate the performance of a ranking algorithm is calculating ranking metrics such as MAP, NDCG. In use cases where there is no ground truth signal, some proxy ...
2
votes
2answers
91 views

inverting the binomial distribution: probability distribution for number of trials necessary to have a given number of successes

The binomial distribution gives me a distribution for the number of successes in several Bernoulli trials, k, given parameters N the total number of trials and q, the success probability for one ...
2
votes
1answer
3k views

KL divergence calculation

I am wondering that how one can calculate KL-divergence on two probability distributions. For example, if we have ...
2
votes
1answer
93 views

Explaining the steps of a visualization tool

this is my first post on CrossValidated. I've done, for academic purpose, a web tool doing this process: web scraping from various sites pre-process the responses (cleaning, error and redundance ...
2
votes
2answers
208 views

Weighting words based on position in text

I'm currently working on semantic analysis and had a question about text organization and structure. Are there any algorithms, or statistical / machine-learning models that weight the importance of a ...
2
votes
1answer
464 views

Clustering structured data: Assessing the similarity of documents that appear in tree structure

Usually when performing text document clustering, similarities across documents are assessed based on the lexical content of documents. But, in my problem, I wish to consider both the lexical content ...
2
votes
2answers
1k views

Clustering of documents that are very different in number of words

I have a corpus of 643 documents with different sizes and my goal is to cluster them according their topics and label each cluster with semantic name for its main topic. I have tired different ...
2
votes
1answer
647 views

How to find “similar documents” after a Latent Dirichlet Allocation model is built

Let's say I run an LDA model with 3 topics on 5 documents. After the model is learned (with Gibbs sampling presumably), I have topic distribution for each document, shown as the following: My ...
2
votes
1answer
1k views

Where did sublinear tf-idf originate?

I have often come across this weighting scheme for tf-idf (term frequency - inverse document frequency) in text mining. I am wondering where it came from (for citations). I've searched very rigorously,...
2
votes
1answer
1k views

Average precision when not all the relevant documents are found

I can't find on the Internet a proper source that explains this. I have built a search engine that for a particular query retrieves 5 relevant document out of the 10 relevant documents. When I ...
2
votes
1answer
1k views

What is the difference between accuracy and agreement?

According to Manning et al. (p. 155) accuracy is the sum of the diagonal in the confusion matrix divided by the sum of all items. On the other hand, following Artstein and Poesio (p . 558) precisely ...
2
votes
0answers
9 views

Models for ranking possible classifications by confidence

What are good means of finding the various highest confidences or likelihoods for a (say) hundred possible outcomes of a classification problem? Inputs belong to only one class, unlike document ...
2
votes
0answers
112 views

How to adjust your dataset so it fits a power-law? [closed]

I have created a dataset of pictures taken at the museum of different paintings. The dataset is divided into 113 different categories (paintings) and contains around 4.8k images. Just to be clear: ...
2
votes
0answers
84 views

F-measure and hypothesis testing

I would compare two classifiers (A and B), where B is obtained from A. I would exploit the F-measure computed on two samples (of two independent populations, p1 and p2, respectively): A -> F-...
2
votes
0answers
524 views

Is there an algorithm for determining scoring function (Utility function) in ranking instances?

So while going through the topic of Preference learning, I came to know about "instance ranking". Since the problem which I'm working on requires me to rank the instances (data point), is there any ...