Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

learn more… | top users | synonyms

1
vote
0answers
18 views

How to prepare a dataset for text classification

I would like to compare some algorithms for performing sentiment classification (Naive Bayes, SVM, and ...
0
votes
0answers
19 views

Using Naive Bayes as vectorizer and SVM for classification in Java [on hold]

0 down vote favorite I am trying to classify legal case documents which are in text format, in different folders like Civil, Land, Criminal, e.t.c, I intended using Naive Bayes as Vectoriser to get ...
0
votes
0answers
24 views

VW multiclass classification

I am new to vw and trying to do a multiclass text classification with 18 classes. features are unigram, bigram and trigrams. Total features are around 1.4 million Total training examples 35 million ...
2
votes
0answers
12 views

How to measure how 'well' I am matching Google keywords?

For google keywords you can bid on a broad match. For example let's say I bid on the keyword 'best hamburger' and somebody searches 'What sort of beef makes the best hamburger?' and 'eat best ...
0
votes
1answer
45 views

In natural language processing (NLP), how do you make an efficient dimension reduction?

In NLP, it's always the case that the dimension of the features are very huge. For example, for one project at hand, the dimension of features is almost 20 thousands (p = 20,000), and each feature is ...
1
vote
0answers
13 views

Calculate association between text documents [duplicate]

I've got 6000 reports. For each report, I've got a vector with the keywords in it and a vector with the tokens of its abstract. Now I want to calculate some association between those two sets. Is ...
-1
votes
0answers
10 views

Viterbi for text segmentation

i have a question concern text segmentation to MWE (tokens) if a i have for example these two sentence: 1- " i read new york times" 2- " the population of india is equal to the population of new york ...
0
votes
1answer
31 views

Vector Space Model for Online News Clustering

I am trying to automatically cluster news articles based on their content. I need this algorithm to be online and simply group news articles related to the same story as they arrive. The common ...
1
vote
0answers
25 views

Has the reported state-of-the-art performance of using paragraph vectors for sentiment analysis been replicated?

I was impressed by the results in the ICML 2014 paper "Distributed Representations of Sentences and Documents" by Le and Mikolov. The technique they describe, called "paragraph vectors", learns ...
0
votes
1answer
14 views

Easy way of indicating the likelihood of a text?

I am trying to find out a simple way of quantifying "how much sense a text paragraph makes". The measure is not necessarily fail-proof, but it has to be somewhat simple. One example measure that ...
3
votes
1answer
64 views

Methodology for standardizing names

We have a great amount of data with user generated names for company names (think of bills where the company wrote their own name). In our data cleaning phase, we clustered the different companies ...
0
votes
0answers
21 views

Dataset with mixed structured/unstructured data

I am looking for some mixed-type dataset with free-form textual features along with some structural features (preferably continuous) - labeled, for supervised setting (classification or prediction). ...
0
votes
2answers
59 views

How to transform test set to the training set transformed space with PCA?

I'm working on a text classification project, and I want to reduce the tf-idf matrix dimension with Principal Component Analysis (PCA) and then train my model with this, which is pretty ...
3
votes
0answers
16 views

TF-IDF Matrix and Regression

I am trying to build a regression model based on some tweets that my company put on our company feed. I would like to transform all of the tweets, and use them to tell me which word(s) were most ...
1
vote
0answers
13 views

Is Social Network Analysis or NER the best way to create a semantic graph?

I am planning to create a semantic graph by creating an automatic ontology. I want to know which is the best process to do it. Doing social network analysis to create people, relationships, likes, ...
1
vote
0answers
84 views

Any advice on how to improve my accuracy rate in text classification?

I'm trying to do a text classification task. Here are some specs: Context file size = 1M+ documents already labeled Number of top-labels = 17 Number of sub-labels = around 130 Each document is ...
0
votes
0answers
20 views

Latent Semantic Analysis: scale representation of documents?

After performing SVD on the term-document matrix, the right eigenvectors correspond to the representation of documents in the reduced concept space. In order to use this for say text classification, ...
0
votes
1answer
112 views

How to use k-fold cross validation in naive bayes classifier?

I'm trying to classify text using naive bayes classifier, and also want to use k-fold cross validation to validate the result of classification. But I'm still confused how to use the k-fold cross ...
0
votes
0answers
27 views

R Clustering Evaluation (Adaptive Kmeans)

i know there are several threads about this topic, but most i read, most i get confused. I'm doing a project that consists in clustering some data (news articles). I used adaptive Kmeans ...
3
votes
1answer
107 views

can we generate a random words from English letters that follow the bigram of the English language

The main issue is that several research building their solution of detecting and classifying English language is based on bigram distribution. However, I would like to know if it possible to generate ...
1
vote
0answers
51 views

In Kneser-Ney smoothing, how are unseen words handled?

From what I have seen, the (second-order) Kneser-Ney smoothing formula is in some way or another given as $ \begin{align} P^2_{KN}(w_n|w_{n-1}) &= \frac{\max \left\{ C\left(w_{n-1}, w_n\right) - ...
1
vote
0answers
72 views

Naive Bayes and text classification: which probability model and vectorizer combination makes sense?

I am wondering which combinations of Naive models can be paired with different vectorizing methods so that it makes sense. Let's say we have a simple binary spam-classification task. Multinomial ...
0
votes
0answers
20 views

choosing best value for N when using N-Gram approach

the question is quite general, but I am doing a research related to supervised machine learning to classify two set of characters into two categories. in fact, I want to compute some measures of ...
0
votes
1answer
84 views

Using topic words generated by LDA to represent a document

I want to do document classification by representing each document as a set of features. I know that there are many ways: BOW, TFIDF, ... I want to use Latent Dirichlet Allocation (LDA) to extract ...
0
votes
0answers
46 views

Unsupervised machine learning with numerical and text data

My dataset has numerical variables and one "Note" variable with around a paragraph of text. I'd like to classify the data with an unsupervised learning algorithm. I've done some searching and one ...
0
votes
0answers
20 views

Asking tweet classification

I want to ask you the process to classify the tweet data. Now, I am working to Twitter data but i have confuse how to classify the tweet data using Mallet Tool. Example; I have 200,000 tweets. The ...
1
vote
0answers
21 views

Extracting city name from free text?

I'm having a set of free text from web. Since the users type their location in that field, we have many un-normalized city names. For example, "Shanghai, China" "China, ShangHai" might mean the ...
0
votes
0answers
14 views

How do I extract a keyword plus its corresponding number out of a text

I am finding my way around in the different text mining tools, but I can't find the technique that does the following: Extract out of the follwing advertisement texts ...
0
votes
1answer
48 views

Using a Dictionary for Text Mining

I want to use a dictionary to analyze a text with text mining mechanisms. So I'm looking for a dictionary with scores for each words, so for example the word "cool", +1, gets a positive score, the ...
0
votes
0answers
13 views

Mallet API installation

I am trying to use the mallet API to classify text. Previously created classifier using the command line tool. I don't know to use Mallet packages using Java. Thank you
0
votes
0answers
5 views

Get positions/values from heat costs bill

for a project I need to extract values from customers yearly heat costs bill. The customer takes a photo of the bill and the program should extract the values heating period of the billing, type of ...
1
vote
0answers
27 views

CoreNLP SemanticGraph - search for edges with specific lemmas

I'm using Stanford CoreNLP's dependency parser, and wondering how to make a generic search for SemanticEdge(s) with specific head lemma, dependent lemma, and lexical relationship. For example, if I ...
0
votes
0answers
8 views

automatic assign class name based on text

My question is , I have a set of plain text , i want to create category based on the text. Eg: i have written something about Soup recepie then the algorithm must create a category called Food. After ...
0
votes
1answer
83 views

Multi-class Confusion Matrix to Binary confusion matrix

i know the main concepts of data/text mining but i used them mainly in binary classification problems (just two classes). i am now dealing with a problem with 8 classes and i am atruggling how to ...
0
votes
0answers
63 views

Using relative frequency for Euclidean and cosine distance (dissimilarity)

How to calculate the Euclidean distance (dissimilarity) between two documents, e.g., D1 and D2 using relative frequency? Here is an example of both cosine and Euclidean distance between two ...
2
votes
0answers
27 views

Finding related words

I have several files, each of which contains unique terms which are related to each other(without sentence structure). So for finding the word relationships I created a dictionary of bi-grams for ...
2
votes
2answers
78 views

Weighting words based on position in text

I'm currently working on semantic analysis and had a question about text organization and structure. Are there any algorithms, or statistical / machine-learning models that weight the importance of a ...
0
votes
0answers
22 views

Categorizing text into IPTC subjects

Does anyone know where I can find a corpus I can use to train a classifier into IPTC news categories (http://www.iptc.org/site/NewsCodes/) ? A google search was not very useful. Thank you
0
votes
1answer
35 views

predict category by using K-NN algorithm having text features

I would like to predict the category of the provided data by using K-NN algorithm. Here is an example of the training data set ...
2
votes
2answers
47 views

Dimensionality reduction (PCA) for plotting text documents on a graph

I have 50 text documents There are 500 possible words, after a stop list has been applied My term/document sparse matrix is therefore 50x500 I'd like to cluster these documents. One easy way to do ...
1
vote
1answer
59 views

SVM Classification with Duplicate Training Instances

I'm using SVMs with linear kernel for sentence classification (binary). My dataset contains many duplicate instances i.e. many sentences in the training set have identical feature vectors. In the ...
0
votes
0answers
25 views

Post-process the output of a Multinomial Naive Bayes text classifier

I have a multinomial text classification application where there are other features than the words in text which can be useful to do the classification e.g, contains email address, contains an URL, ...
3
votes
3answers
167 views

In Naive Bayes, why bother with Laplacian smoothing when we have unknown words in the test set?

I was reading over Naive Bayes Classification today. I read, under the heading of Parameter Estimation with add 1 smoothing: "Let $c$ refer to a class (such as Positive or Negative), and let $w$ ...
1
vote
0answers
29 views

Power Analysis for Text Mining

I have a population of 6 million text files with which I want to perform sentiment analysis (and text analysis more generally). I will need to manually hand code a subset of these texts into positive ...
1
vote
0answers
33 views

Binary classification of dated text documents with seasonality

I have a collection of training documents with publication dates, where each document is labeled as belonging (or not) to some topic T. I want to train a model that will predict for a new document ...
0
votes
0answers
94 views

What algorithms should I use to perform job classification based on resume data?

Note that I am doing everything in R. The problem goes as follow: Basically, I have a list of resumes (CVs). Some candidates will have work experience before and some don't. The goal here is to: ...
1
vote
1answer
349 views

Feature selection : how to select the Information Gain threshold?

I am trying to use Information Gain to select features when classifying text with a Support Vector Machine. For each word in our training data, we computed its information gain. Then, we should keep ...
0
votes
1answer
31 views

How to increase a particular terms's weightage?

I am doing Text classification using LibSVM in Rapid Miner. I am using TFIDF values for processing documents. I need to Increase weightage of some terms in the documents(for eg. words in BOLD and ...
0
votes
0answers
30 views

How to collect related words with specified one?

Assume I have a word such as statistics, and research articles or text books about statistics. My question is how to detect top 100 related, highly associated ...
0
votes
0answers
135 views

How to implement data I have to svmtrain() function in MATLAB?

I have to write a script using MATLAB which will classify my data. My data consists of 1051 web pages (rows) and 11000+ words (columns). The first 230 rows are about computer science course (to be ...