Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

0
votes
0answers
2 views

Combine Noisy predictions from an optical character recognition program

I am trying to perform optical character recognition on a field of text from different angles as the camera passes over it and beyond it. Due to the 3D skewing of the image, thee readings of the OCR ...
0
votes
0answers
6 views

MDS for word vectors: Weighting and distance measures

I calculated a Latent Dirichlet Allocation (LDA) based customer reviews for three different smartphone brands. The topics represent the attributes of the brands. I want to map the single topics on an ...
0
votes
0answers
15 views

How to denote token length in statistics

I have a vocabulary (V) that includes terms. (e.g., V = ['data mining', 'data', 'machine', 'machine learning']) I also have a ...
1
vote
1answer
18 views

Get association of categorical variables

I do a text analysis where I want to identify dependencies among categorical variables, for example let's take this dataset: ...
0
votes
0answers
6 views

Using LDA to create two layered wordcloud

I create a word-cloud using LDA model what I want to do is to find the documents IDs related to that topic group. So, for example, the image here I want to allow users to click the word broccoli and ...
0
votes
0answers
8 views

Correlations in text analysis - for an absolute beginner

I am analysing the media releases and speeches of two political leaders. I want to find out: how often they mention young people, and in what context. I know my frequency tables are correct because I ...
0
votes
0answers
4 views

How PV-DBOW works

The authors of the Paragraph Vector paper describe PV-DBOW with: 2.3. Paragraph Vector without word ordering: Distributed bag of words The above method considers the concatenation of the ...
0
votes
1answer
33 views

How to find correlation between text data?

I have data set similar to this: I want to if the columns subtype and item are correlated. They have different text, ...
0
votes
0answers
12 views

How to solve the estimate population characters with sample text-mining?

I've recently learning text-mining, but none of my textbooks talk of inferential statistics; they talk about how to analyze collected data but hardly deal with how to estimate population data through ...
0
votes
0answers
5 views

wordfish - must reference texts be at the absolute extreme of the spectrum

I have a question pertaining to the wordfish method (see here for the academic paper introducing the package and here for the package itself). My question regards ...
0
votes
0answers
20 views

How to statistically infer common pattern in text

Am trying to solve a problem where I need to infer common patterns in text for example, the data below, with bare eyes it can be noticed there is a pattern and that is ...
0
votes
0answers
6 views

Calculating the ratio of the (frequency of a specific word in the corpus/sum of the frequency of all the words)

I have the following code which gives me the list of top 10 words in a corpus in descending order of frequency: library(tidytext) tidybooks.nstop<-tidy_books %>% + anti_join(stop_words) ...
0
votes
0answers
13 views

Classification of sample with only unseen words

I'm doing text classification (Product Name) where one example belongs to one class. "Some Product Name" -> MODEL -> {CLASS_1 | CLASS_2 | CLASS_3 | CLASS_4} ...
2
votes
0answers
66 views

How to compute gain statistic for the multinomial Naive Bayes classifier from Jurafsky and Martin (2018)

I'm trying to figure out how to compute the gain statistic G(w) following the fitting of the multinomial Naive Bayes model. This statistic is described on p17 of the new edition of Jurafsky and ...
0
votes
0answers
25 views

Sentiment Analysis Issues Using R

I am using the R package sentimentr and sentiment function to get polarity/sentiment scores on a list of comments. The issue I am having is that the comments are in ...
0
votes
0answers
14 views

Difference between Semantic analysis and Syntactic analysis in text analysis

Can someone explain me clearly the meaning of Semantic analysis and Syntactic Analysis in text analysis ? I am very much confused between these two. Also tell the difference in how they are used in ...
0
votes
0answers
14 views

How to ensemble predictions from image classifier and text classifier?

I am doing multiclass classification based on images and text. I have predictions from both image classification and text. I am not sure how to combine them. Should I use probabilities as a feature to ...
0
votes
0answers
16 views

Guided LDA to categorize software requirements

I'm developing a application to categorize requirements in a requirement specification in to categories like database, front end, back end, etc. So for that I'm trying to use Guided LDA since labelled ...
0
votes
0answers
24 views

Pointwise Mutual Information Word Dependency

I have pre-defined concepts which are either a single word or couple of words that refer to a concept.( In the context of machine learning for instance, covariance matrix is a concept). I am trying to ...
2
votes
0answers
15 views

Correlation of phrases and answers to them

If I ask question in wrong forum, let me know, I'll delete it. I try find methods, methodology to create predicative model. Namely, I want to investigate the relationship between the sentences and ...
0
votes
0answers
15 views

naive bayes text classification: can I look at the individual word probabilities?

Assume I use the Naive Bayes classification algorithm. My question is simple: can I rank the words according to their posterior probabilities? I want to have a measure of "importance" of the words in ...
1
vote
1answer
11 views

Good way to use word similarity as a feature in supervised ML on text

I have a pretty low N data set of small sentences tagged with a label. I would like to create a classifier on this dataset. The word choice is not very variable since the domain is pretty specific. ...
0
votes
0answers
20 views

Using a priori knowledge in a classification task

I'm working on a classification task, related with text classification, where texts to be classified are requests for technical support, and the classes are technical guys which issues can be assigned ...
1
vote
1answer
39 views

Finding similar text - algorithms and evaluation

I've been asked to create a program that will rank similar texts to an input text given a collection of text. So far I've been using a tdidf representation and cosine similarity with a lot of regex-...
0
votes
0answers
26 views

Getting the document-per-topic loading using TextmineR package by passing term co-occurrence matrix

I am using TextmineR package to find the most similar documents to given document list. I used the following code to generate the tcm not dtm ...
0
votes
1answer
18 views

apply CountVectorized to whole data before applying train_test_split

Is there any difference between the two different snips of codes. ...
0
votes
0answers
15 views

Required sample size for sentiment analysis in customer satisfaction

Suppose we are interested in seeing how companies A through F are being viewed by different customers, and how this changes based on the customers' positions. A sample of customers is drawn from ...
1
vote
0answers
15 views

Information theoretic alternative to tf-idf heuristic?

I've been recently working with feature construction from texts, where tf-idf measure is one of the main options for vectorizing the documents (one feature per e.g., word). I was wondering, whether ...
0
votes
0answers
15 views

What is the best model for keyphrase extraction from super long text?

I’m working on a keyphrase extraction task. The biggest difficulty of this task is that the text is very long (5000-20000 words). I’ve tried several unsupervised algorithms such as Tf-idf and TextRank ...
0
votes
1answer
308 views

Imbalanced multiclass classification with many classes

I am working on a text classification project in which we have hundreds of (imbalanced) classes. Some characteristics of the data: We have examples of "bad" documents. Basically documents that don't ...
0
votes
0answers
34 views

Applying Label From Supervised Learning to Unlabeled Data- Text Classification

I am wondering if anyone has code to following: 1) Apply labels from a previous text classification dataset like this type of data (https://colab.research.google.com/drive/...
0
votes
1answer
48 views

text preprocessing using keras [closed]

I am getting started with NLP, in kaggle , and it dont get how this keras preprocessing works if anyone could explain the code would be much helpful,thanks ...
1
vote
0answers
131 views

Clustering as a method to find and label classes for supervised learning

I'm working on a text classification project. We have around 300k documents (small, 1~2 phrases) and we don't know the set of labels or how many labels there are. The recommended approach to me was ...
0
votes
0answers
241 views

Can cosine similarity be used to measure similarity between words?

In text mining books, I generally see cosine similarity used as a way to assess the similarity in documents; however, by transposing a tf-idf matrix, one can also calculate cosine similarity between ...
2
votes
1answer
294 views

Why does Naive Bayes work better when the number of features >> sample size compared to more sophisticated ML algorithms?

According to this article Because of the class independence assumption, naive Bayes classifiers can quickly learn to use high dimensional features with limited training data compared to more ...
1
vote
0answers
20 views

What is an appropriate Evaluation Metric and corresponding Loss function which best optimize the metric for a classification based FAQ Chatbot?

I am developing a FAQ chatbot to display/return only one correct answer in a chat window for a given question from the user. I know MRR & MAP make sense as an evaluation metric for information ...
0
votes
1answer
81 views

Applying PCA - First two components explain low variance but have high data separation when plotting

Applying PCA on a set of documents gives strange results in terms of the variance explained by the PCs vs the data separation I'm having when plotting the first two principle components. Details: ...
1
vote
1answer
55 views

How to implement topic modelling in regression analysis

I have a dataset consisting of hotel reviews, ratings, and other features such as traveller type, and word count of the review. I want to perform topic modeling (LDA) and use the topics derived from ...
0
votes
1answer
54 views

Text Similarity - Cosine - Control. Suggestion to another / better method?

I would like to ask you, if anybody could check my code, because it was behaving weird - not working, giving me errors to suddenly working without changing anything - the code will be at the bottom. ...
0
votes
0answers
17 views

Unsupervised learning with DNN on text

I want to: Have a key:pair database with author:largetextfileofeverysentencetheauthorpublished.txt Set up a deep neural network to see without supervision patterns in choice of vocabulary. Have the ...
0
votes
1answer
36 views

Comparison of text data for same distribution

I have two datasets with different text in it. I want to check if they are from the same distribution. If they were numbers tabulated in some way, it would be easy. Since this is text, how can this be ...
0
votes
0answers
108 views

Current SOTA in text classification

I've been recently starting with text classification, yet the amount of work in this field is somewhat overwhelming. Could you please direct me towards some of the SOTA (deep) approaches for solving ...
0
votes
0answers
13 views

Enrich true signal from loads of data

Background For simplicity let's say we have an alphabet of ABC and we are looking at words that all have the same length (n = 10)...
1
vote
0answers
38 views

HDP: Gibbs sampler implementation

I am trying to recreate the model proposed by Gao et al. (2011), based on the Hierarchical Dirichlet Process proposed by Teh and al. (2005). To estimate the model (let's call it iHDP) I need to ...
0
votes
0answers
56 views

How do I normalize or weight document-feature-matrix by length of dictionary entries

What is the best practice way to normalize or weight document-feature-matrices by the length of dictionary entries. Here is some sample code. In reality, my example works with different issues e.g. ...
7
votes
4answers
3k views

Check if a character string is not random

Background Let's say we have an alphabet of A,B, C, D, then we look through some data and find a "word" which is DDDDDDDDCDDDDDD ...
0
votes
0answers
43 views

How do document lengths affect Gaussian Naive Bayes?

I'm trying to understand Gaussian Naive Bayes. I am training on a pre-processed subset of the 20 Newsgroup data. Each observation is around 500 attributes (words), and 1 class (of 5 possible). I ...
2
votes
1answer
424 views

Word2Vec and PyTorch - am I approaching this correctly?

My understanding of Word2Vec is that the library allows for generation of an array of numbers that approximates the meaning of a word relative to others in a sentence. My use of Word2Vec e.g. ...
0
votes
1answer
59 views

Clustering short messages

I have a dataset of short message conversations (from 1 to 20 words). I would like to cluster the messages that were sent to me to extract the different topics that were discussed by my interlocutors. ...
0
votes
1answer
11 views

text mining - vocabulary size very large

Question: when you have create a corpus of let’s say, 10,000 documents, and the vocabulary size made for these is let’s say, 1 million, what best practices exist to either work with this type of ...