0
votes
0answers
18 views

choosing best value for N when using N-Gram approach

the question is quite general, but I am doing a research related to supervised machine learning to classify two set of characters into two categories. in fact, I want to compute some measures of ...
2
votes
2answers
78 views

Weighting words based on position in text

I'm currently working on semantic analysis and had a question about text organization and structure. Are there any algorithms, or statistical / machine-learning models that weight the importance of a ...
0
votes
0answers
15 views

Can LSA find correlations between multiple words?

I need to find correlations between multiple terms (say, 3 or 4) in a single-term search index. I'm trying to figure out if LSA fits to the problem. Am I right that LSA is no more than a term-to-term ...
0
votes
0answers
23 views

maximize mean F1 score in multilabel information retrieval problem

I have a multilabel text classification problem where each observation will have one or more labels associated to it. The metric I want to maximize is mean F1 score. Are there standard ways to ...
1
vote
1answer
36 views

How do I perform an IDF calculation?

How do I perform an IDF calculation? I am uncertain as to whether IDF should be calculated in per-class level or for the entire document set (that contains multiple classes).
0
votes
0answers
31 views

What are the mathematics I need to learn, before I start research in data mining [duplicate]

I usually use text mining, graph mining, Information retrieval, and natural lanuage processing. Also i will use the fundamental concepts of data mining like classification, association and clustering. ...
0
votes
0answers
45 views

Build corpus with phrases

I have my documents as: doc1 = beautifull, very good, very bad, you are great doc2 = very bad, good restaurent, nice place to visit I want to make my corpus ...
1
vote
1answer
140 views

Create a matrix of tf-idf values from documents

I have a set of documents like: D1 = "The sky is blue." D2 = "The sun is bright." D3 = "The sun in the sky is bright." and a ...
0
votes
1answer
70 views

Clustering structured data: Assessing the similarity of documents that appear in tree structure

Usually when performing text document clustering, similarities across documents are assessed based on the lexical content of documents. But, in my problem, I wish to consider both the lexical content ...
1
vote
0answers
64 views

Learning to Rank: query-dependent vs. query-independent features

I've been doing some reading about learning to rank - specifically lambdaMART - and one thing I am confused about is the role of features. When training a model, should one only use query-dependent ...
0
votes
1answer
57 views

Automatic labeling of training set

I have once meet the following question, given a training set, is that possible to do the automatic labelling? In addition, if this training set consists of plain text files, is that possible to know ...
2
votes
2answers
126 views

Dataset and papers for baseline [closed]

I'm doing a project about Topic Detection and Tracking in text. I need to perform a baseline so I can compare existing results with mine. I read some papers where they use datasets that are not so ...
1
vote
2answers
131 views

Most important journals in data mining/ML, NLP and IR?

Can you please provide with me with the names of the most important journals in data mining, machine learning, natural language processing and information retrieval?
0
votes
1answer
230 views

K-Fold Cross validation and F1 Measure Score for Document Retrieval using TF-IDF weighting and some customised weighting schemes

I am developing a search engine system based on the vector space model, and I am confused on what approach I should take to evaluate the system. My case is this: I have a set of indexed documents ...
2
votes
2answers
244 views

Other documents features than tf-idf for clustering?

What are other feature representations for documents that are used for clustering textual documents? The only representation I'm aware of is tf-idf. Are there other ones?
3
votes
1answer
460 views

A parellel between LSA and pLSA

In the original paper of pLSA the author, Thomas Hoffman, draw a parallel between pLSA and LSA data structures that I would like to discuss with you. Background: Taking inspiration the Information ...
1
vote
0answers
30 views

Evaluation and Testsets for NNMF

I am trying to evaluate my recommender system which uses Non-negative Matrix Factorization. Some things that I evaluate are How does the size of the feature matrix affect the recommendations How ...
2
votes
2answers
305 views

How do you predict a continuous value from many booleans & a continuous value?

Hello: I am a computer science student working as a research assistant in an undergrad IR lab, feeling spectacularly out of my element. Given an input of a single continuous value and a vector of ...
2
votes
0answers
82 views

Software library for Hidden Markov Modeling of a large text database

Given we have a large database of texts (e.g. product descriptions) and we want to extract multiple types of information (e.g. brand, release date, features, price, etc.) what's a good library to ...
7
votes
2answers
317 views

Understanding and applying sentiment analysis

I was just having been assigned a project of conducting sentiment analysis for some document collections. By Googling, a lot of sentiment-related research has popped up. My questions are: What are ...
2
votes
2answers
572 views

KL divergence calculation

I am wondering that how one can calculate KL-divergence on two probability distributions. For example, if we have ...