# Tagged Questions

The tag has no usage guidance.

13 views

### Probabilistic Information Retrieval Question

I thought up an interesting information retrieval question that I am having a difficult time answering. Consider a semantic search engine i.e. a search engine that matches on semantically similar ...
23 views

### Normalized term frequency comparisons across documents of differing length & language

I aim to infer on the prevalence of terms across and within corpora of different languages (where document length varies within and across corpora). Given Zipf’s and Heap’s laws a simple tf/n seems ...
24 views

### F-measure and hypothesis testing

I would compare two classifiers (A and B), where B is obtained from A. I would exploit the F-measure computed on two samples (of two independent populations, p1 and p2, respectively): A -> F-...
70 views

### Difference between Log Entropy Model and TF-IDF Model?

I would like to understand what are the differences/advantages in using TF-IDF or the Log Entropy model for represeting documents and queries in an information retrieval system using diferent weights. ...
34 views

### Where did sublinear tf-idf originate?

I have often come across this weighting scheme for tf-idf (term frequency - inverse document frequency) in text mining. I am wondering where it came from (for citations). I've searched very rigorously,...
19 views

### Where do initial document values come from in K-means document clustering?

So the K-means algorithm seems simple enough as I understand it: given some documents, turn those documents into points, initialize some number of k (centroids), assign document-points to nearest ...
12 views

### How to decide what is the relevant group in a precision and recall computation?

One of the most famous measurements for an information retrieval system is to compute its precision and recall. For both cases, we need to compute the number of total relevant documents and compare it ...
24 views

### topic modeling for image retrieval

I'm interested in learning a topic model from a bag of visual words for image retrieval. I can compute V cluster centers (visual words) of SIFT descriptors at keypoints for each training image and fit,...
14 views

### Acronym Resolution Issues

I am trying to work out the resolution of acronyms. Till now I could find two good papers in http://cogprints.org/4399/1/NRC-48078.pdf and http://www.lrec-conf.org/proceedings/lrec2010/pdf/...
45 views

### The Effect of Stopword Filtering prior to Word Embedding Training

Recently I have played with the pretrained GLOVE word embedding model for Twitter http://nlp.stanford.edu/projects/glove/ I notice that common stopwords are existing in the model. That is, there is ...
21 views

### Computing average precision at depth n (AP@n)

I have a binary classification problem, for which I use some kernel SVM (e.g., with the RBF kernel). I apply the trained SVM model to a testing set and I get a list with predictions (labels) along ...
62 views

### How do you compare words or documents using LSA (latent semantic analysis)

As the title says, I am a bit cofunsed in how documents or words are compared using LSA (when I say compare, I am referring to calculate similarities, for instance, cosine similarity). In An ...
41 views

### how to find the nearest neighbor of a sparse vector

I have about 500 vectors,each vector is a 1500-dimension vector, and almost every vector is very sparse-- I mean only about 30-70 dimension of the vector is not 0。 Now, the problom is that here is a ...
62 views

### Is there an algorithm for determining scoring function (Utility function) in ranking instances?

So while going through the topic of Preference learning, I came to know about "instance ranking". Since the problem which I'm working on requires me to rank the instances (data point), is there any ...
8 views

### Query Expansion: Combining relevant doc information given a base query

I am working on a search problem. I have a bunch of short queries such as "fuzzy bunny" as well as some short documents which I know to be related / contain these queries. For example, for "fuzzy ...
18 views

### Extracting document's keyword after dimensionality reduction

Let's say I have a word document matrix and I applied SVD to this matrix (LSA), and now I have the representation of this matrix in a reduced dimensional space. How could I use this for extracting ...
30 views

### Is this ordering of examples redundant in Rank SVM?

I was studying RankSVM[1] and I was observing the ranking pairs listed in the example here. Formulation can be seen on page 4 of this paper. One of the ordering is:...
24 views

### Normalised score for BM25

BM25 provides a function that assigns a score that is a function of a query and a document. The score is computed for each document in the collection and can be used to rank documents, but this score ...
63 views

### Explaining the steps of a visualization tool

this is my first post on CrossValidated. I've done, for academic purpose, a web tool doing this process: web scraping from various sites pre-process the responses (cleaning, error and redundance ...
23 views

### Measuring efficiency of hierarchical clustering alone

I have designed a model which uses hierarchical clustering to predict user habit, by either stating number (0) or number (1) in a specific context. It uses a three stage approach with 4d data (n ...
39 views

### What is the measurement scale of the Reciprocal Rank metric?

The reciprocal rank (RR) metric is often used to measure to what extent ranking approaches are able to return relevant items in the topmost positions of the list of results. It is computed as follows: ...
218 views

### Typical range of values for TFIDF

I am working on a text corpus. Each line contains between 10 and 50 words. There are around 25 000 words in the whole text and 1 000 000 lines. I turned this corpus into its tf-idf representation. I ...
109 views

### Information retrieval from strings using neural network

I am dealing with input strings that have max length of 200 characters, they are the USSD popup msgs and the SMS text strings on android smart phones extracted as and when they appear via an android ...
170 views

171 views

### Is precision in recommender system related to mean average error (MAE)?

A recommender system is being evaluated while increasing the neighborhood size. The highest precision was achieved between 10-15 neighbors(users) while the lowest MAE was in the range from 30-40 users....
101 views

### Search in a graph of medical records

I have a hierarchical order of a disease list. Just for example: Respiratory system disease 1.1. Asphyxia neonatorum 1.2. Croup 1.3. Lower respiratory tract disease 1.4. Bronchial disease 1....
188 views

### Vector Space Model for Online News Clustering

I am trying to automatically cluster news articles based on their content. I need this algorithm to be online and simply group news articles related to the same story as they arrive. The common ...
307 views

### Difference between Weighted Average Entropy and Adjusted Mutual Information (for evaluating Clustering)

I was advised by my team leader to use this weighted average entropy to evaluating the performance of my dbscan clustering algorithm, and its mathematical formulation is: Scikit provides what many ...
52 views

### Measure precision

This is used in information retrieval. I have an algorithm that uses a sample set to predict a yes / no. I think the correct term is binomial. Yes this document is about sports or no this document ...
157 views

### Correct Evaluation of Random Forest on fixed training/test set

I have to test the performance of Random Forest on the same dataset (text classification) with about 118.000 instances of which about 1/3 is used for training and 2/3 is used for testing. The division ...
107 views

### Weighting words based on position in text

I'm currently working on semantic analysis and had a question about text organization and structure. Are there any algorithms, or statistical / machine-learning models that weight the importance of a ...
188 views

### In inverse theory, how do I transform the averaging kernel matrix to a new grid?

Rodgers and Connor (2003) describe how measurements by remote sounders can be properly compared, taking into account differences in averaging kernels and error covariances. They make the assumption ...
17 views

### Information retrieval performance measures for unknown test collection?

I am evaluating a web search relevance feedback algorithm. The algorithm uses Bing API as source for its result sets. To evaluate the algorithm I will be conducting a user study. In the end I will ...
20 views

### Evaluation of IR approach without test collections

I am trying to evaluate an information retrieval approach. More specifically, it's a query expansion algorithm, based on topic distribution in retrieved documents. I want to evaluate my approach ...
48 views

### How do I perform an IDF calculation?

How do I perform an IDF calculation? I am uncertain as to whether IDF should be calculated in per-class level or for the entire document set (that contains multiple classes).
33 views

### What are the mathematics I need to learn, before I start research in data mining [duplicate]

I usually use text mining, graph mining, Information retrieval, and natural lanuage processing. Also i will use the fundamental concepts of data mining like classification, association and clustering. ...
3k views

### Create a matrix of tf-idf values from documents

I have a set of documents like: D1 = "The sky is blue." D2 = "The sun is bright." D3 = "The sun in the sky is bright." and a ...
147 views

### Clustering structured data: Assessing the similarity of documents that appear in tree structure

Usually when performing text document clustering, similarities across documents are assessed based on the lexical content of documents. But, in my problem, I wish to consider both the lexical content ...
305 views

### Learning to Rank: query-dependent vs. query-independent features

I've been doing some reading about learning to rank - specifically lambdaMART - and one thing I am confused about is the role of features. When training a model, should one only use query-dependent ...
51 views

### Choosing the best set of keywords

I have a dataset of tweets collected using twitter streaming API on a particular topic (say 'football') using around 40 keywords. Now if I'm going to track the same topic (football) in future how do I ...