The tag has no usage guidance.

learn more… | top users | synonyms

3
votes
2answers
107 views

Vector Space Model for Online News Clustering

I am trying to automatically cluster news articles based on their content. I need this algorithm to be online and simply group news articles related to the same story as they arrive. The common ...
0
votes
1answer
34 views

Calculation of Cosine Similarity using LSA

I am getting negative cosine similarity value between two documents in Latent Semantic analysis. How should it be treated?
-1
votes
1answer
26 views

Data mining : I want to automate data scrapping process in ruby. Which approach would be better. Supervised or unsupervised?

I've read that the problem with supervised approach is it fails to generalize if the structure of web page changes. But sites rarely change their structure. Problem with unsupervised is it will ...
3
votes
0answers
32 views

Heuristics streaming data matching

I have an index composed by thousands of documents. Slightly modified copies of those documents are sent to my application in small chunks, and I need to check, from those chunks, which document has ...
3
votes
0answers
333 views

Similarity calculations for arrays

First of all, my apologies if I mess up the terminology. I've been out of math for several years, so I'm certain I'm going to use terms incorrectly. Also, though I concentrated mathematics in college, ...
2
votes
0answers
93 views

Search in a graph of medical records

I have a hierarchical order of a disease list. Just for example: Respiratory system disease 1.1. Asphyxia neonatorum 1.2. Croup 1.3. Lower respiratory tract disease 1.4. Bronchial disease ...
2
votes
0answers
31 views

Evaluation of a semi-supervised ranking model

I learned ranking model with graph-based semi-supervised approach, while labeled (just positive) and unlabeled (positive and negative) data is both used in training. With the model, all of the data ...
2
votes
0answers
367 views

Why are ERR (Expected Reciprocal Ranking) scores not normalized?

It seems to me that normalized ERR (Expected Reciprocal Ranking) scores (ERR scores of your ranking algorithm divided by ERR score calculated for the ground truth ranking) are more useful than the ...
2
votes
0answers
95 views

Software library for Hidden Markov Modeling of a large text database

Given we have a large database of texts (e.g. product descriptions) and we want to extract multiple types of information (e.g. brand, release date, features, price, etc.) what's a good library to ...
2
votes
0answers
173 views

How to to calculate the topic distribution of a document

I have a simple (may be stupid) question. I want to calculate KullbackÔÇôLeibler divergence on two documents. It requires probability distribution of each document. I do not know how to calculate ...
1
vote
0answers
141 views

Difference between Weighted Average Entropy and Adjusted Mutual Information (for evaluating Clustering)

I was advised by my team leader to use this weighted average entropy to evaluating the performance of my dbscan clustering algorithm, and its mathematical formulation is: Scikit provides what many ...
1
vote
0answers
15 views

Information retrieval performance measures for unknown test collection?

I am evaluating a web search relevance feedback algorithm. The algorithm uses Bing API as source for its result sets. To evaluate the algorithm I will be conducting a user study. In the end I will ...
1
vote
0answers
141 views

Learning to Rank: query-dependent vs. query-independent features

I've been doing some reading about learning to rank - specifically lambdaMART - and one thing I am confused about is the role of features. When training a model, should one only use query-dependent ...
1
vote
0answers
77 views

Mutual Information for clustering

I'm working on a document clustering application and decided to use Normalized Mutual Information as one of the measures of effectivenes. But I don't really understand how to implement this in that ...
1
vote
0answers
66 views

MAE/MSE with or without square root

I read some papers about recommender systems and information retrieval, where Mean Absolut Error and Mean Squared Error are mentioned. But I've found some differences between the formal definition of ...
1
vote
0answers
102 views

Is that overfitting?

Given a document-term matrix $X$, where $$X(d, t) = \textit{occurrences of 't' in 'd'}$$, it's possible to compute it's Truncated Singular Value Decomposition:$$X_k = U_k \Sigma_k V_k^T$$ Then, for a ...
1
vote
0answers
144 views

Using sentiment lexicons or all words processing for sentiment analysis?

I am learning sentiment analysis to apply it to twitter real time data to predict user's mood. I ponder about using which alternative way to do that data mining job. Use all words to process and ...
1
vote
0answers
34 views

Evaluation and Testsets for NNMF

I am trying to evaluate my recommender system which uses Non-negative Matrix Factorization. Some things that I evaluate are How does the size of the feature matrix affect the recommendations How ...
0
votes
0answers
10 views

Measuring efficiency of hierarchical clustering alone

I have designed a model which uses hierarchical clustering to predict user habit, by either stating number (0) or number (1) in a specific context. It uses a three stage approach with 4d data (n ...
0
votes
0answers
10 views

What is the measurement scale of the Reciprocal Rank metric?

The reciprocal rank (RR) metric is often used to measure to what extent ranking approaches are able to return relevant items in the topmost positions of the list of results. It is computed as follows: ...
0
votes
0answers
23 views

Typical range of values for TFIDF

I am working on a text corpus. Each line contains between 10 and 50 words. There are around 25 000 words in the whole text and 1 000 000 lines. I turned this corpus into its tf-idf representation. I ...
0
votes
0answers
31 views

Information retrieval from strings using neural network

I am dealing with input strings that have max length of 200 characters, they are the USSD popup msgs and the SMS text strings on android smart phones extracted as and when they appear via an android ...
0
votes
0answers
26 views

An IR evalualuation metric that only measures the rank of results?

I am working on a little text clustering problem, and trying to figure out how to evaluate the results. I came up with the following idea that I though fits pretty well with the specifics of the ...
0
votes
0answers
37 views

Understanding the Precision-Recall breakeven point in ranking

Consider an IR system that retrieves all document and rank them in order of relevance. Now, we can calculate Precision and Recall at each rank. Does the breakeven point (recall=precision) always ...
0
votes
0answers
34 views

Test statistics in IR (t-test vs Wilcoxon vs signed test)

I have just read this paper about different test statistics one could use in Information Retrieval (IR), discussing whether Wilcoxon, Sign-test or Student t-test would be more reliable for comparing ...
0
votes
0answers
17 views

Can using of Weight function in Latent Semantic Analysis increase the efficiency of the technique?

What type of combination of local & global weight function and what would be the length of the document which can actually increase the efficiency of the LSA.
0
votes
0answers
18 views

Evaluating annotations with gold standard

I want to evaluate the following: I have a gold set with documents wherefore the meta data is annotated; this means for an article i have annotated what the author is, what the title is, etc.. I now ...
0
votes
0answers
13 views

Calculating parameters for a custom ranking algorithm with no training data

On my site, we want to be able to sort search results for books by hot they are. When a book is added to the site its hotness is set to number of hours since the epoch, purchases, reviews and ...
0
votes
0answers
66 views

Are there differences between Delta TF-IDF and TF-IDF?

Are there differences between both algorithm or not ? i mean if i implement for ex Delta TF-IDF in a project instead of ...
0
votes
0answers
23 views

Picking a right classification metric

I have a classification problem that I ended up with only true positives (TP). Now, I would like to measure my classification but I wonder what kind of metric I should pick for the evaluation. Let ...
0
votes
0answers
64 views

Is precision in recommender system related to mean average error (MAE)?

A recommender system is being evaluated while increasing the neighborhood size. The highest precision was achieved between 10-15 neighbors(users) while the lowest MAE was in the range from 30-40 ...
0
votes
0answers
14 views

parsing semi-structured textual data poorly matrix formatted

I have found many approaches in the literature that could deal with my problem (which is parsing poorly formatted textual data having a matrix structure but whose content may vary, headers being ...
0
votes
0answers
10 views

Modeling 2-D data as vector

As part of my class project i had to use decision tree classification on a training set which contains a set of matrices where each row is a vector recorded at a particular time stamp and each row ...
0
votes
0answers
12 views

Context-aware information retrieval?

I was asked a puzzle lately and so I decided to cheat and find the answer on Google. The puzzle was asked to me in a different language than English, so I had to think about the main context and then ...
0
votes
0answers
47 views

choosing best value for N when using N-Gram approach

the question is quite general, but I am doing a research related to supervised machine learning to classify two set of characters into two categories. in fact, I want to compute some measures of ...
0
votes
0answers
20 views

Evaluation of IR approach without test collections

I am trying to evaluate an information retrieval approach. More specifically, it's a query expansion algorithm, based on topic distribution in retrieved documents. I want to evaluate my approach ...
0
votes
0answers
51 views

Calculation of normalization constant

This is an equation from the paper "A Content-based Probabilistic Correction Model for OCR Document Retrieval" - Rong Jin, Alex G. Hauptmann , ChengXiang Zhai $$P(w|M_{\text{orig}})= ...
0
votes
0answers
47 views

Algorithms for keyphrase clustering

Are there any standard algorithms for keyphrase clustering. There are several algorithms for keyphrase extraction from a corpus. For e.g. this publication reviews some of the popular keyphrase ...