The tag has no usage guidance.

learn more… | top users | synonyms

0
votes
1answer
20 views

how to find the nearest neighbor of a sparse vector

I have about 500 vectors,each vector is a 1500-dimension vector, and almost every vector is very sparse-- I mean only about 30-70 dimension of the vector is not 0。 Now, the problom is that here is a ...
0
votes
1answer
69 views

Calculation of Cosine Similarity using LSA

I am getting negative cosine similarity value between two documents in Latent Semantic analysis. How should it be treated?
3
votes
0answers
47 views

Heuristics streaming data matching

I have an index composed by thousands of documents. Slightly modified copies of those documents are sent to my application in small chunks, and I need to check, from those chunks, which document has ...
3
votes
0answers
344 views

Similarity calculations for arrays

First of all, my apologies if I mess up the terminology. I've been out of math for several years, so I'm certain I'm going to use terms incorrectly. Also, though I concentrated mathematics in college, ...
2
votes
0answers
96 views

Search in a graph of medical records

I have a hierarchical order of a disease list. Just for example: Respiratory system disease 1.1. Asphyxia neonatorum 1.2. Croup 1.3. Lower respiratory tract disease 1.4. Bronchial disease ...
2
votes
0answers
224 views

Learning to Rank: query-dependent vs. query-independent features

I've been doing some reading about learning to rank - specifically lambdaMART - and one thing I am confused about is the role of features. When training a model, should one only use query-dependent ...
2
votes
0answers
34 views

Evaluation of a semi-supervised ranking model

I learned ranking model with graph-based semi-supervised approach, while labeled (just positive) and unlabeled (positive and negative) data is both used in training. With the model, all of the data ...
2
votes
0answers
488 views

Why are ERR (Expected Reciprocal Ranking) scores not normalized?

It seems to me that normalized ERR (Expected Reciprocal Ranking) scores (ERR scores of your ranking algorithm divided by ERR score calculated for the ground truth ranking) are more useful than the ...
2
votes
0answers
98 views

Software library for Hidden Markov Modeling of a large text database

Given we have a large database of texts (e.g. product descriptions) and we want to extract multiple types of information (e.g. brand, release date, features, price, etc.) what's a good library to ...
2
votes
0answers
179 views

How to to calculate the topic distribution of a document

I have a simple (may be stupid) question. I want to calculate Kullback–Leibler divergence on two documents. It requires probability distribution of each document. I do not know how to calculate ...
1
vote
0answers
30 views

Is there an algorithm for determining scoring function (Utility function) in ranking instances?

So while going through the topic of Preference learning, I came to know about "instance ranking". Since the problem which I'm working on requires me to rank the instances (data point), is there any ...
1
vote
0answers
22 views

Is this ordering of examples redundant in Rank SVM?

I was studying RankSVM[1] and I was observing the ranking pairs listed in the example here. Formulation can be seen on page 4 of this paper. One of the ordering ...
1
vote
0answers
17 views

Information retrieval performance measures for unknown test collection?

I am evaluating a web search relevance feedback algorithm. The algorithm uses Bing API as source for its result sets. To evaluate the algorithm I will be conducting a user study. In the end I will ...
1
vote
0answers
87 views

Mutual Information for clustering

I'm working on a document clustering application and decided to use Normalized Mutual Information as one of the measures of effectivenes. But I don't really understand how to implement this in that ...
1
vote
0answers
68 views

MAE/MSE with or without square root

I read some papers about recommender systems and information retrieval, where Mean Absolut Error and Mean Squared Error are mentioned. But I've found some differences between the formal definition of ...
1
vote
0answers
107 views

Is that overfitting?

Given a document-term matrix $X$, where $$X(d, t) = \textit{occurrences of 't' in 'd'}$$, it's possible to compute it's Truncated Singular Value Decomposition:$$X_k = U_k \Sigma_k V_k^T$$ Then, for a ...
1
vote
0answers
157 views

Using sentiment lexicons or all words processing for sentiment analysis?

I am learning sentiment analysis to apply it to twitter real time data to predict user's mood. I ponder about using which alternative way to do that data mining job. Use all words to process and ...
1
vote
0answers
34 views

Evaluation and Testsets for NNMF

I am trying to evaluate my recommender system which uses Non-negative Matrix Factorization. Some things that I evaluate are How does the size of the feature matrix affect the recommendations How ...
0
votes
0answers
12 views

How do you compare words or documents using LSA (latent semantic analysis)

As the title says, I am a bit cofunsed in how documents or words are compared using LSA (when I say compare, I am referring to calculate similarities, for instance, cosine similarity). In An ...
0
votes
0answers
8 views

Query Expansion: Combining relevant doc information given a base query

I am working on a search problem. I have a bunch of short queries such as "fuzzy bunny" as well as some short documents which I know to be related / contain these queries. For example, for "fuzzy ...
0
votes
0answers
13 views

Extracting document's keyword after dimensionality reduction

Let's say I have a word document matrix and I applied SVD to this matrix (LSA), and now I have the representation of this matrix in a reduced dimensional space. How could I use this for extracting ...
0
votes
0answers
12 views

Normalised score for BM25

BM25 provides a function that assigns a score that is a function of a query and a document. The score is computed for each document in the collection and can be used to rank documents, but this score ...
0
votes
0answers
22 views

Measuring efficiency of hierarchical clustering alone

I have designed a model which uses hierarchical clustering to predict user habit, by either stating number (0) or number (1) in a specific context. It uses a three stage approach with 4d data (n ...
0
votes
0answers
23 views

What is the measurement scale of the Reciprocal Rank metric?

The reciprocal rank (RR) metric is often used to measure to what extent ranking approaches are able to return relevant items in the topmost positions of the list of results. It is computed as follows: ...
0
votes
0answers
94 views

Typical range of values for TFIDF

I am working on a text corpus. Each line contains between 10 and 50 words. There are around 25 000 words in the whole text and 1 000 000 lines. I turned this corpus into its tf-idf representation. I ...
0
votes
0answers
76 views

Information retrieval from strings using neural network

I am dealing with input strings that have max length of 200 characters, they are the USSD popup msgs and the SMS text strings on android smart phones extracted as and when they appear via an android ...
0
votes
0answers
28 views

An IR evalualuation metric that only measures the rank of results?

I am working on a little text clustering problem, and trying to figure out how to evaluate the results. I came up with the following idea that I though fits pretty well with the specifics of the ...
0
votes
0answers
80 views

Understanding the Precision-Recall breakeven point in ranking

Consider an IR system that retrieves all document and rank them in order of relevance. Now, we can calculate Precision and Recall at each rank. Does the breakeven point (recall=precision) always ...
0
votes
0answers
66 views

Test statistics in IR (t-test vs Wilcoxon vs signed test)

I have just read this paper about different test statistics one could use in Information Retrieval (IR), discussing whether Wilcoxon, Sign-test or Student t-test would be more reliable for comparing ...
0
votes
0answers
21 views

Can using of Weight function in Latent Semantic Analysis increase the efficiency of the technique?

What type of combination of local & global weight function and what would be the length of the document which can actually increase the efficiency of the LSA.
0
votes
0answers
27 views

Evaluating annotations with gold standard

I want to evaluate the following: I have a gold set with documents wherefore the meta data is annotated; this means for an article i have annotated what the author is, what the title is, etc.. I now ...
0
votes
0answers
20 views

Calculating parameters for a custom ranking algorithm with no training data

On my site, we want to be able to sort search results for books by hot they are. When a book is added to the site its hotness is set to number of hours since the epoch, purchases, reviews and ...
0
votes
0answers
120 views

Are there differences between Delta TF-IDF and TF-IDF?

Are there differences between both algorithm or not ? i mean if i implement for ex Delta TF-IDF in a project instead of ...
0
votes
0answers
107 views

Is precision in recommender system related to mean average error (MAE)?

A recommender system is being evaluated while increasing the neighborhood size. The highest precision was achieved between 10-15 neighbors(users) while the lowest MAE was in the range from 30-40 ...
0
votes
0answers
20 views

Evaluation of IR approach without test collections

I am trying to evaluate an information retrieval approach. More specifically, it's a query expansion algorithm, based on topic distribution in retrieved documents. I want to evaluate my approach ...
0
votes
0answers
52 views

Calculation of normalization constant

This is an equation from the paper "A Content-based Probabilistic Correction Model for OCR Document Retrieval" - Rong Jin, Alex G. Hauptmann , ChengXiang Zhai $$P(w|M_{\text{orig}})= ...
0
votes
0answers
52 views

Algorithms for keyphrase clustering

Are there any standard algorithms for keyphrase clustering. There are several algorithms for keyphrase extraction from a corpus. For e.g. this publication reviews some of the popular keyphrase ...