The tag has no wiki summary.

learn more… | top users | synonyms

1
vote
1answer
60 views

Vector Space Model for Online News Clustering

I am trying to automatically cluster news articles based on their content. I need this algorithm to be online and simply group news articles related to the same story as they arrive. The common ...
0
votes
1answer
10 views

Calculation of Cosine Similarity using LSA

I am getting negative cosine similarity value between two documents in Latent Semantic analysis. How should it be treated?
3
votes
0answers
28 views

Heuristics streaming data matching

I have an index composed by thousands of documents. Slightly modified copies of those documents are sent to my application in small chunks, and I need to check, from those chunks, which document has ...
3
votes
0answers
329 views

Similarity calculations for arrays

First of all, my apologies if I mess up the terminology. I've been out of math for several years, so I'm certain I'm going to use terms incorrectly. Also, though I concentrated mathematics in college, ...
2
votes
0answers
91 views

Search in a graph of medical records

I have a hierarchical order of a disease list. Just for example: Respiratory system disease 1.1. Asphyxia neonatorum 1.2. Croup 1.3. Lower respiratory tract disease 1.4. Bronchial disease ...
2
votes
0answers
28 views

Evaluation of a semi-supervised ranking model

I learned ranking model with graph-based semi-supervised approach, while labeled (just positive) and unlabeled (positive and negative) data is both used in training. With the model, all of the data ...
2
votes
0answers
264 views

Why are ERR (Expected Reciprocal Ranking) scores not normalized?

It seems to me that normalized ERR (Expected Reciprocal Ranking) scores (ERR scores of your ranking algorithm divided by ERR score calculated for the ground truth ranking) are more useful than the ...
2
votes
0answers
90 views

Software library for Hidden Markov Modeling of a large text database

Given we have a large database of texts (e.g. product descriptions) and we want to extract multiple types of information (e.g. brand, release date, features, price, etc.) what's a good library to ...
2
votes
0answers
165 views

How to to calculate the topic distribution of a document

I have a simple (may be stupid) question. I want to calculate KullbackÔÇôLeibler divergence on two documents. It requires probability distribution of each document. I do not know how to calculate ...
1
vote
0answers
14 views

Information retrieval performance measures for unknown test collection?

I am evaluating a web search relevance feedback algorithm. The algorithm uses Bing API as source for its result sets. To evaluate the algorithm I will be conducting a user study. In the end I will ...
1
vote
0answers
107 views

Learning to Rank: query-dependent vs. query-independent features

I've been doing some reading about learning to rank - specifically lambdaMART - and one thing I am confused about is the role of features. When training a model, should one only use query-dependent ...
1
vote
0answers
74 views

Mutual Information for clustering

I'm working on a document clustering application and decided to use Normalized Mutual Information as one of the measures of effectivenes. But I don't really understand how to implement this in that ...
1
vote
0answers
63 views

MAE/MSE with or without square root

I read some papers about recommender systems and information retrieval, where Mean Absolut Error and Mean Squared Error are mentioned. But I've found some differences between the formal definition of ...
1
vote
0answers
97 views

Is that overfitting?

Given a document-term matrix $X$, where $$X(d, t) = \textit{occurrences of 't' in 'd'}$$, it's possible to compute it's Truncated Singular Value Decomposition:$$X_k = U_k \Sigma_k V_k^T$$ Then, for a ...
1
vote
0answers
135 views

Using sentiment lexicons or all words processing for sentiment analysis?

I am learning sentiment analysis to apply it to twitter real time data to predict user's mood. I ponder about using which alternative way to do that data mining job. Use all words to process and ...
1
vote
0answers
30 views

Evaluation and Testsets for NNMF

I am trying to evaluate my recommender system which uses Non-negative Matrix Factorization. Some things that I evaluate are How does the size of the feature matrix affect the recommendations How ...
0
votes
0answers
19 views

An IR evalualuation metric that only measures the rank of results?

I am working on a little text clustering problem, and trying to figure out how to evaluate the results. I came up with the following idea that I though fits pretty well with the specifics of the ...
0
votes
0answers
7 views

Understanding the Precision-Recall breakeven point in ranking

Consider an IR system that retrieves all document and rank them in order of relevance. Now, we can calculate Precision and Recall at each rank. Does the breakeven point (recall=precision) always ...
0
votes
0answers
18 views

Test statistics in IR (t-test vs Wilcoxon vs signed test)

I have just read this paper about different test statistics one could use in Information Retrieval (IR), discussing whether Wilcoxon, Sign-test or Student t-test would be more reliable for comparing ...
0
votes
0answers
13 views

Can using of Weight function in Latent Semantic Analysis increase the efficiency of the technique?

What type of combination of local & global weight function and what would be the length of the document which can actually increase the efficiency of the LSA.
0
votes
0answers
11 views

Evaluating annotations with gold standard

I want to evaluate the following: I have a gold set with documents wherefore the meta data is annotated; this means for an article i have annotated what the author is, what the title is, etc.. I now ...
0
votes
0answers
7 views

Calculating parameters for a custom ranking algorithm with no training data

On my site, we want to be able to sort search results for books by hot they are. When a book is added to the site its hotness is set to number of hours since the epoch, purchases, reviews and ...
0
votes
0answers
39 views

Are there differences between Delta TF-IDF and TF-IDF?

Are there differences between both algorithm or not ? i mean if i implement for ex Delta TF-IDF in a project instead of ...
0
votes
0answers
16 views

Picking a right classification metric

I have a classification problem that I ended up with only true positives (TP). Now, I would like to measure my classification but I wonder what kind of metric I should pick for the evaluation. Let ...
0
votes
0answers
45 views

Is precision in recommender system related to mean average error (MAE)?

A recommender system is being evaluated while increasing the neighborhood size. The highest precision was achieved between 10-15 neighbors(users) while the lowest MAE was in the range from 30-40 ...
0
votes
0answers
12 views

parsing semi-structured textual data poorly matrix formatted

I have found many approaches in the literature that could deal with my problem (which is parsing poorly formatted textual data having a matrix structure but whose content may vary, headers being ...
0
votes
0answers
9 views

Modeling 2-D data as vector

As part of my class project i had to use decision tree classification on a training set which contains a set of matrices where each row is a vector recorded at a particular time stamp and each row ...
0
votes
0answers
11 views

Context-aware information retrieval?

I was asked a puzzle lately and so I decided to cheat and find the answer on Google. The puzzle was asked to me in a different language than English, so I had to think about the main context and then ...
0
votes
0answers
95 views

Difference between Weighted Average Entropy and Adjusted Mutual Information (for evaluating Clustering)

I was advised by my team leader to use this weighted average entropy to evaluating the performance of my dbscan clustering algorithm, and its mathematical formulation is: Scikit provides what many ...
0
votes
0answers
38 views

choosing best value for N when using N-Gram approach

the question is quite general, but I am doing a research related to supervised machine learning to classify two set of characters into two categories. in fact, I want to compute some measures of ...
0
votes
0answers
5 views

Get positions/values from heat costs bill

for a project I need to extract values from customers yearly heat costs bill. The customer takes a photo of the bill and the program should extract the values heating period of the billing, type of ...
0
votes
0answers
106 views

Using relative frequency for Euclidean and cosine distance (dissimilarity)

How to calculate the Euclidean distance (dissimilarity) between two documents, e.g., D1 and D2 using relative frequency? Here is an example of both cosine and Euclidean distance between two ...
0
votes
0answers
14 views

Search for similar documents on growing term-base

Could you please recommend any approach (if it exists) for similar document search considering the following: It's necessary to provide term base explicitly. This is a dictionary of specific named ...
0
votes
0answers
24 views

Can LSA find correlations between multiple words?

I need to find correlations between multiple terms (say, 3 or 4) in a single-term search index. I'm trying to figure out if LSA fits to the problem. Am I right that LSA is no more than a term-to-term ...
0
votes
0answers
17 views

Evaluation of IR approach without test collections

I am trying to evaluate an information retrieval approach. More specifically, it's a query expansion algorithm, based on topic distribution in retrieved documents. I want to evaluate my approach ...
0
votes
0answers
38 views

maximize mean F1 score in multilabel information retrieval problem

I have a multilabel text classification problem where each observation will have one or more labels associated to it. The metric I want to maximize is mean F1 score. Are there standard ways to ...
0
votes
0answers
152 views

Build corpus with phrases

I have my documents as: doc1 = beautifull, very good, very bad, you are great doc2 = very bad, good restaurent, nice place to visit I want to make my corpus ...
0
votes
0answers
50 views

Calculation of normalization constant

This is an equation from the paper "A Content-based Probabilistic Correction Model for OCR Document Retrieval" - Rong Jin, Alex G. Hauptmann , ChengXiang Zhai $$P(w|M_{\text{orig}})= ...
0
votes
0answers
44 views

Algorithms for keyphrase clustering

Are there any standard algorithms for keyphrase clustering. There are several algorithms for keyphrase extraction from a corpus. For e.g. this publication reviews some of the popular keyphrase ...