Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

learn more… | top users | synonyms

0
votes
0answers
5 views

Latent dirichlet allocation Variational inference in R

I am trying to do analyse text by using Lda variational inference method. I built document term matrix which is from newspaper. I wonder if anybody know how to code up in R to analyse . if you can ...
0
votes
0answers
18 views

Will feature reduction help classification, regardless of the algorithm?

I have a data set (originating from text) with p=4000-10000 features (words and/or concepts) and n=2000-4000 obervations. My target value is binary (true/false). I apply different ML algorithms for ...
1
vote
1answer
23 views

bayesian classification unknown domain

Suppose I am building a naive Bayes model to classify text messages as either spam or legit. I am training my model using a dataset containing both classes and for which I know the domain (the number ...
1
vote
0answers
46 views

Latent Semantic Analysis- Theme extraction [on hold]

I have a large collection of text documents I am trying to group using Singular Value Decomposition for dimension reduction(popularly known as LSA) followed by k-means clustering. To be able to ...
0
votes
0answers
17 views

How to do sentence boundary detection? [closed]

I have a string as My name is sushil feeling bad My age is 30 and hobbies are none thanks for asking me name varun i am good at cricket and my age is 40 i have done a lot of work in research area , ...
2
votes
0answers
45 views

Is Robert Rutledge's calculation of the probability of similarities between Michelle Obama and Melania Trump's speeches reasonable?

Melania Trump has given a speech in support of her husband's campaign that many people believe has been plagiarised. An astrophysics professor, Robert Rutledge has calculated that there is a 1 in 87 ...
0
votes
0answers
8 views

Speech Analytics relating text mining

Any good app which could convert speech into text?i have some call data which i need to analyse.There i need an application which would convert speech into text and then perform text mining on the ...
-3
votes
0answers
24 views

text mining in crime from online newspaper [closed]

i am working on a project called text mining on crime from online news paper in English and i need a perfect road map can you please tell me the right way to do this and i m trying to develop an ...
0
votes
0answers
11 views

Proper Document to Term Ratio for LDA

In preparing my DTM for LDA (latent Dirichlet allocation, not linear discriminate analysis) in R, I do the usual cleansing process of converting to lowercase, stemming, removal of stopwords, non alpha-...
1
vote
2answers
71 views

How to categorize (lookup) brands with spelling mistakes

I have a dataset with around 4 million rows. The data has only column which contains around 50000 unique brand names (e.g. IBM, Google, Adobe, Microsoft etc.). I also have a lookup table which ...
0
votes
0answers
27 views

Outlier/Anomaly detection in strings

Can anyone suggest methods/techniques to find anomaly in string data from database. The data contains road names, so every cell is unique. By outlier, I mean the strings with weird stuff(special ...
0
votes
0answers
28 views

Smart sampling techniques to ensure boundary/edge cases are included

In an effort to reduce the time it takes to test software against various data sets I would like to do create an application that looks at a data set and creates a "smart sample" from the it. The ...
0
votes
0answers
14 views

Classification of text & numeric data in the same dataset

I have more of a methodology question rather than a technical issue. I have a somewhat odd classification problem (at least to my limited knowledge). I have a a file with several numeric columns (...
0
votes
0answers
18 views

knn text clasiffication model error when term is not found in new documents

Using KNN model for topic clasification. My model uses 200 variables (terms) and 10 target labels, using R (tm package). Accuracy is fairly good. Now, new documents are arriving that need to be ...
3
votes
1answer
103 views

Was it as valid to perform k-means on a distance matrix as on data matrix (text mining data)?

(This post is a repost of a question I posted yesterday (now deleted), but I've tried to scale back volume of words and simplify what I'm asking) I'm hoping to get some help interpreting a kmeans ...
0
votes
0answers
14 views

Finding common, evenly present theme (words) across documents

I have a collection of six documents, each of which is a bag of words (not sentences, just words) that describes a certain category of product. I'm trying to figure out the common binding "theme" ...
0
votes
0answers
11 views

Document Length Normalization

What is the difference between document length (total no of words in the document) normalization and cosine normalization? How do we usually normalize a document represented as a set of words and ...
0
votes
0answers
26 views

How to pre-process data & tune hyperparameters of Doc2Vec?

I am using doc2vec for getting document similarity (unsupervised learning). I read that we need to shuffle the input matrix to doc2vec & reduce the learning rate for better performance. But the ...
0
votes
0answers
15 views

How to processing a document to extract the description of certain properties of a reference domain?

I should analyze a text in order to identify the description of certain properties of some objects belonging to a given reference domain. The objects and their properties are known, as well as the ...
1
vote
0answers
53 views

Properties of Levenshtein, N-Gram, cosine and Jaccard distance coefficients - in sentence matching

Let's say I have two strings: string A: 'I went to the cafeteria and bought a sandwich.' string B: 'I heard the cafeteria is serving roast-beef sandwiches today'. ...
0
votes
0answers
16 views

Simple English translations regarding thematic textual analysis

I have run a thematic textual analysis of economic select committee meetings in the UK using t-lab. I ran a thematic analysis of the E.C.U.s, and categorised the E.C.U.s into (for example) 5 thematic ...
0
votes
0answers
9 views

Text classification using sentiment scores and test content?

Suppose there are are two snippets: t1: "Hey! I like this bag 'A' but I hope its color was better." t2: "I like most of colors the bag A comes in." You see that both statements talk about the same ...
1
vote
1answer
12 views

Choosing cases for assisted supervised learning

I have a bag of words binary text classification task. The SGD algorithm performed well for a certain target where number of labeled cases for training reached tens of thousands. For another target ...
1
vote
0answers
14 views

Ordering tdm in specific order, not frequencies

I have this corpus of documents, properly pre-processed. Working with R package tm, i do: ...
0
votes
0answers
24 views

Normalized term frequency comparisons across documents of differing length & language

I aim to infer on the prevalence of terms across and within corpora of different languages (where document length varies within and across corpora). Given Zipf’s and Heap’s laws a simple tf/n seems ...
0
votes
0answers
19 views

Which Deep Learning architecture should I use for generating one summary from different texts that talk about the same topic?

In my problem I want to develop a system capable to generate a summary or fusion of different texts that talk about similar topics (e.g. news articles). I have read about deep learning and for now it ...
0
votes
1answer
28 views

number of features in feature selection for text mining problems

Let's say for a text mining problem (e.g creating a predictive model using text analysis), using a feature selection method (e.g TF-IDF) we come up with 1000 features/words/tokens. Is there some ...
0
votes
0answers
49 views

Latent Dirichlet Allocation for Twitter dataset

I'm trying to understand Latent Dirichlet Allocation (LDA) to apply on Twitter dataset. I've a dataset with 10k tweets and I've already splitted tweets in six groups. Now I'd extract topic from each ...
0
votes
0answers
38 views

How to interpret the results of clustering on text documents

I am working on Text Analysis of the Feedback's given in a Survey. I wanted to identify the different themes or topics people are talking about. So, i have desired to go ahead and do Clustering. ...
1
vote
0answers
12 views

Data analysis approach for structured data [closed]

I have large chunk of structured data. It is mainly issues/bugs, requirements submitted on a web, stored in a database. e.g. data will be organized as Title, ID, Subject detail, issue ingredient, ...
1
vote
0answers
34 views

Where did sublinear tf-idf originate?

I have often come across this weighting scheme for tf-idf (term frequency - inverse document frequency) in text mining. I am wondering where it came from (for citations). I've searched very rigorously,...
0
votes
0answers
25 views

Linear model coefficients with text mining

Suppose that I have a collection of reviews about food - perhaps reviews for a restaurant or something. Also suppose that I'm interested in predicting the score from these reviews. One approach that I ...
0
votes
1answer
28 views

TF IDF Analysis for tag extraction

So I have used the tm package to perform TF IDF analysis on around 1000 documents. What I want to do is to extract common tags from those documents. At the moment I already have a matrix where the ...
0
votes
0answers
32 views

Why getting better classification results despite many irrelevant terms?

I am new to ML especially for document (text) classification. I have 22 classes (scientific fields) and I am trying to improve classification results by employing some additional data. That is, I use ...
1
vote
0answers
55 views

Pros and Cons: LDA vs Neural Networks

LDA is an older approach for word representations, there are newer methods now like CBOW and Skip-gram. But what are the improvements of these models? Do they improve in every way or does LDA still ...
1
vote
0answers
30 views

Selecting correct settings for the order of Minkowski distance

I am looking to compute the distance between vectors of word frequencies (and I am new to this). I am trying out the Minkowski distance as implemented in Scipy. The documentation asks me to specify a "...
0
votes
0answers
21 views

How large should be negative train set in text classification of rare category

I'm working on the classification of medical texts in order to find texts about the quite rare disease from the big set of all medical articles. I have the set of positive examples and some negative ...
1
vote
0answers
19 views

Getting significantly different results for a classification task when using two similar approaches

I am new to machine learning and I classify abstracts of scientific papers retrieved from two different disciplines. I use RTextTools package and I apply two different approaches for the ...
0
votes
0answers
29 views

Text mining to find the significant date in a news article

Lets say we have a group of news articles that have already been classified as pertaining to an event (such as a conference or public announcment). The last step in the problem is to determine what ...
0
votes
0answers
43 views

What is the meaning of laplace, eps and threshold in NaiveBayes package in R e1071 lib?

I am using NaiveBayes for text classification, I am interested on tagging a text (like a blog post). What I am finding is that normally I have results in which a tag has a probability of 0.9999 of ...
0
votes
0answers
17 views

How to assess text classification results when extreme sparsity is present?

I try to classify documents based on bag-of-words single word approach. I employ R with its RTextTools to use SVM. My text files are like below: TRAIN ...
1
vote
2answers
88 views

How to use MaxEnt (logistic regression) weights?

I asked this question last night and Matt Krause explanation helped me a lot. (For more explanation please see my previous question). Now I have another problem. We are using RapidMiner studio and ...
1
vote
1answer
66 views

Low recall and high precision in text summarization

We are trying to generate a model to summarize Persian news. About 14000 news were summarized with help of humans(supervised) and then we extracted all sentences (about 180000) and labeled them (...
1
vote
1answer
41 views

Latent Semantic Analysis: stop words and link words

On many tutorials about how to implement LSA, I see that stop words such as "and" are removed. I understand that we might find them in almost all kind of texts, but the repetition of link words in a ...
0
votes
0answers
9 views

Train Textual Pattern Detector

I'm trying to train a model that can detect structures of text (and maybe label them in case of multiple structures) in a coprus. Exemple: Train a model with a dataset of addresses: ...
0
votes
0answers
30 views

text mining (document term matrix)

I am doing a text mining project in R using "tm" package . I have successfully built a document term matrix. I want to remove terms having frequency >75 % and terms having frequency <25 % ...
1
vote
1answer
200 views

What is text distance in data mining

I need to write a report on visualization of multidimensional data, map and text distance. I got content related to other two but not getting any clue about text distance. Is it related to Data ...
1
vote
1answer
42 views

Text classification algorithms for small sets

I'm trying to classify a set of 1656 tweets into different categories. I've read about different classification algorithms (supervised and unsupervised) but I'm really concerned because my set and ...
1
vote
0answers
18 views

Is there any package in R for Sound like analysys of text [closed]

The words "Jhon" and "Joan" may sound similar although spelling is different. Is there any package for "Sound like Analysis" in R
0
votes
1answer
31 views

Doubt about feature selection

I'm working on a text classification problem using Python and NLTK. I've got two frequency distributions, one for each class (it's basically a binary classification). So, my doubt it's if there's a ...