Tagged Questions

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

learn more… | top users | synonyms

0
votes
0answers
3 views

Query() function within the 'lsa' package [on hold]

Does anybody have a link or an example to how one can input a query (e.g. a string of text or comma-separated words) and return a collection of matching documents by ranked "score" / return documents ...
0
votes
0answers
28 views

How to transform test set to the training set transformed space with PCA?

I'm working on a text classification project, and I want to reduce the tf-idf matrix dimension with Principal Component Analysis (PCA) and then train my model with this, which is pretty ...
0
votes
0answers
18 views

Text mining conferences [closed]

What are the most significant text mining conferences? Please include one conference per answer. Please also include URL link for the conference.
-1
votes
0answers
14 views

How to create a feature vector with scikit-learn? [closed]

I have extracted some bigrams from a corpus, how can i create a feature vector with those bigrams with scikit-learn?, could anybody provide me some example?. Thanks
3
votes
0answers
12 views

TF-IDF Matrix and Regression

I am trying to build a regression model based on some tweets that my company put on our company feed. I would like to transform all of the tweets, and use them to tell me which word(s) were most ...
1
vote
0answers
12 views

Is Social Network Analysis or NER the best way to create a semantic graph?

I am planning to create a semantic graph by creating an automatic ontology. I want to know which is the best process to do it. Doing social network analysis to create people, relationships, likes, ...
1
vote
0answers
70 views

Any advice on how to improve my accuracy rate in text classification?

I'm trying to do a text classification task. Here are some specs: Context file size = 1M+ documents already labeled Number of top-labels = 17 Number of sub-labels = around 130 Each document is ...
0
votes
0answers
20 views

Latent Semantic Analysis: scale representation of documents?

After performing SVD on the term-document matrix, the right eigenvectors correspond to the representation of documents in the reduced concept space. In order to use this for say text classification, ...
0
votes
1answer
58 views

How to use k-fold cross validation in naive bayes classifier?

I'm trying to classify text using naive bayes classifier, and also want to use k-fold cross validation to validate the result of classification. But I'm still confused how to use the k-fold cross ...
0
votes
0answers
20 views

R Clustering Evaluation (Adaptive Kmeans)

i know there are several threads about this topic, but most i read, most i get confused. I'm doing a project that consists in clustering some data (news articles). I used adaptive Kmeans ...
3
votes
1answer
64 views

can we generate a random words from English letters that follow the bigram of the English language

The main issue is that several research building their solution of detecting and classifying English language is based on bigram distribution. However, I would like to know if it possible to generate ...
1
vote
0answers
29 views

In Kneser-Ney smoothing, how are unseen words handled?

From what I have seen, the (second-order) Kneser-Ney smoothing formula is in some way or another given as $ \begin{align} P^2_{KN}(w_n|w_{n-1}) &= \frac{\max \left\{ C\left(w_{n-1}, w_n\right) - ...
1
vote
0answers
37 views

Naive Bayes and text classification: which probability model and vectorizer combination makes sense?

I am wondering which combinations of Naive models can be paired with different vectorizing methods so that it makes sense. Let's say we have a simple binary spam-classification task. Multinomial ...
0
votes
0answers
18 views

choosing best value for N when using N-Gram approach

the question is quite general, but I am doing a research related to supervised machine learning to classify two set of characters into two categories. in fact, I want to compute some measures of ...
0
votes
1answer
69 views

Using topic words generated by LDA to represent a document

I want to do document classification by representing each document as a set of features. I know that there are many ways: BOW, TFIDF, ... I want to use Latent Dirichlet Allocation (LDA) to extract ...
0
votes
0answers
45 views

Unsupervised machine learning with numerical and text data

My dataset has numerical variables and one "Note" variable with around a paragraph of text. I'd like to classify the data with an unsupervised learning algorithm. I've done some searching and one ...
0
votes
0answers
18 views

Asking tweet classification

I want to ask you the process to classify the tweet data. Now, I am working to Twitter data but i have confuse how to classify the tweet data using Mallet Tool. Example; I have 200,000 tweets. The ...
1
vote
0answers
18 views

Extracting city name from free text?

I'm having a set of free text from web. Since the users type their location in that field, we have many un-normalized city names. For example, "Shanghai, China" "China, ShangHai" might mean the ...
0
votes
0answers
14 views

How do I extract a keyword plus its corresponding number out of a text

I am finding my way around in the different text mining tools, but I can't find the technique that does the following: Extract out of the follwing advertisement texts ...
0
votes
1answer
33 views

Using a Dictionary for Text Mining

I want to use a dictionary to analyze a text with text mining mechanisms. So I'm looking for a dictionary with scores for each words, so for example the word "cool", +1, gets a positive score, the ...
0
votes
0answers
13 views

Mallet API installation

I am trying to use the mallet API to classify text. Previously created classifier using the command line tool. I don't know to use Mallet packages using Java. Thank you
0
votes
0answers
5 views

Get positions/values from heat costs bill

for a project I need to extract values from customers yearly heat costs bill. The customer takes a photo of the bill and the program should extract the values heating period of the billing, type of ...
1
vote
0answers
24 views

CoreNLP SemanticGraph - search for edges with specific lemmas

I'm using Stanford CoreNLP's dependency parser, and wondering how to make a generic search for SemanticEdge(s) with specific head lemma, dependent lemma, and lexical relationship. For example, if I ...
0
votes
0answers
8 views

automatic assign class name based on text

My question is , I have a set of plain text , i want to create category based on the text. Eg: i have written something about Soup recepie then the algorithm must create a category called Food. After ...
0
votes
1answer
62 views

Multi-class Confusion Matrix to Binary confusion matrix

i know the main concepts of data/text mining but i used them mainly in binary classification problems (just two classes). i am now dealing with a problem with 8 classes and i am atruggling how to ...
0
votes
0answers
50 views

Using relative frequency for Euclidean and cosine distance (dissimilarity)

How to calculate the Euclidean distance (dissimilarity) between two documents, e.g., D1 and D2 using relative frequency? Here is an example of both cosine and Euclidean distance between two ...
2
votes
0answers
26 views

Finding related words

I have several files, each of which contains unique terms which are related to each other(without sentence structure). So for finding the word relationships I created a dictionary of bi-grams for ...
2
votes
2answers
78 views

Weighting words based on position in text

I'm currently working on semantic analysis and had a question about text organization and structure. Are there any algorithms, or statistical / machine-learning models that weight the importance of a ...
0
votes
0answers
18 views

Categorizing text into IPTC subjects

Does anyone know where I can find a corpus I can use to train a classifier into IPTC news categories (http://www.iptc.org/site/NewsCodes/) ? A google search was not very useful. Thank you
0
votes
1answer
30 views

predict category by using K-NN algorithm having text features

I would like to predict the category of the provided data by using K-NN algorithm. Here is an example of the training data set ...
2
votes
2answers
43 views

Dimensionality reduction (PCA) for plotting text documents on a graph

I have 50 text documents There are 500 possible words, after a stop list has been applied My term/document sparse matrix is therefore 50x500 I'd like to cluster these documents. One easy way to do ...
1
vote
1answer
56 views

SVM Classification with Duplicate Training Instances

I'm using SVMs with linear kernel for sentence classification (binary). My dataset contains many duplicate instances i.e. many sentences in the training set have identical feature vectors. In the ...
0
votes
0answers
22 views

Post-process the output of a Multinomial Naive Bayes text classifier

I have a multinomial text classification application where there are other features than the words in text which can be useful to do the classification e.g, contains email address, contains an URL, ...
3
votes
3answers
111 views

In Naive Bayes, why bother with Laplacian smoothing when we have unknown words in the test set?

I was reading over Naive Bayes Classification today. I read, under the heading of Parameter Estimation with add 1 smoothing: "Let $c$ refer to a class (such as Positive or Negative), and let $w$ ...
1
vote
0answers
27 views

Power Analysis for Text Mining

I have a population of 6 million text files with which I want to perform sentiment analysis (and text analysis more generally). I will need to manually hand code a subset of these texts into positive ...
1
vote
0answers
31 views

Binary classification of dated text documents with seasonality

I have a collection of training documents with publication dates, where each document is labeled as belonging (or not) to some topic T. I want to train a model that will predict for a new document ...
0
votes
0answers
76 views

What algorithms should I use to perform job classification based on resume data?

Note that I am doing everything in R. The problem goes as follow: Basically, I have a list of resumes (CVs). Some candidates will have work experience before and some don't. The goal here is to: ...
1
vote
1answer
194 views

Feature selection : how to select the Information Gain threshold?

I am trying to use Information Gain to select features when classifying text with a Support Vector Machine. For each word in our training data, we computed its information gain. Then, we should keep ...
0
votes
1answer
27 views

How to increase a particular terms's weightage?

I am doing Text classification using LibSVM in Rapid Miner. I am using TFIDF values for processing documents. I need to Increase weightage of some terms in the documents(for eg. words in BOLD and ...
0
votes
0answers
29 views

How to collect related words with specified one?

Assume I have a word such as statistics, and research articles or text books about statistics. My question is how to detect top 100 related, highly associated ...
0
votes
0answers
92 views

How to implement data I have to svmtrain() function in MATLAB?

I have to write a script using MATLAB which will classify my data. My data consists of 1051 web pages (rows) and 11000+ words (columns). The first 230 rows are about computer science course (to be ...
0
votes
0answers
49 views

Hierarchical text classification in R

I am working Automatic Email routing. My historical data set has Email Description,Division and Category. There are around 20 division and 150 categories. I have built one vs. all classifier ...
2
votes
3answers
201 views

Naive Bayes: Imbalanced Dataset in Real-time Scenario

I am using scikit-learn Multinomial Naive Bayes classifier for binary text classification (classifier tells me whether the document belongs to the category X or not). I use a balanced dataset to train ...
1
vote
1answer
35 views

Assigning meaningful cluster name automatically

The objective of my work is to cluster the text documents. Once the documents are clustered, traditionally the system will assign numeric value for the clustered group. For example if I have 5 ...
1
vote
1answer
36 views

How do I perform an IDF calculation?

How do I perform an IDF calculation? I am uncertain as to whether IDF should be calculated in per-class level or for the entire document set (that contains multiple classes).
0
votes
1answer
57 views

I have an easy Text mining frequency question

I have a very simple question. I am doing a simple frequency analysis where I am looking at two documents and comparing the frequency of the top used words in each document. I want to do a comparison ...
2
votes
1answer
51 views

Selecting number of clustering classes automatically

I am working in text clustering. I would like to find a way to identify the number of classes for the clustering process automatically rather than proving the number of class manually. Is their any ...
0
votes
0answers
31 views

What are the mathematics I need to learn, before I start research in data mining [duplicate]

I usually use text mining, graph mining, Information retrieval, and natural lanuage processing. Also i will use the fundamental concepts of data mining like classification, association and clustering. ...
1
vote
1answer
138 views

Comparing topic distributions between corpora using Latent Dirichlet Allocation and R topicmodels or python gensim

So I am working on a problem where I want to extract a set of LDA topics from one corpus, and then compare the distribution of those topics in other corpora. So basically I want to lock-in the topics ...
1
vote
1answer
153 views

Create a matrix of tf-idf values from documents

I have a set of documents like: D1 = "The sky is blue." D2 = "The sun is bright." D3 = "The sun in the sky is bright." and a ...