Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
0
votes
0answers
18 views

Clean News article in NLP [closed]

I collected news dataset for the fake detection task. The dataset consists of many text files in which each text file contains a news article (news story). These articles may have useless ...
1
vote
0answers
7 views

Recommendation Engine and Text Analytics

I am looking for a dataset on which I can use Collaborative filtering and Content based filtering along with Text Mining. Could anybody please suggest , is there any dataset on which I can apply ...
0
votes
0answers
31 views

Estimating Required Data for NLP Classification Models

Are there general guidelines for how much data is required for natural language processing (NLP) classification models? I understand this may depend on the text quality, text length, how accurate the ...
1
vote
0answers
17 views

Variable selection, variable reduction, and handling sparsity for binary text classification

I am trying to do a binary text classification using support vector machine. I am wondering if I am doing it right and I'd like to look for some answers to the questions in mind. The following ...
1
vote
1answer
34 views

Text classification with small dataset for a specialized domain

I have a multiclass text classification problem where I have very few documents for each class. The classes are imbalanced but I want to be able to predict the class when I have at least 200 - 300 ...
0
votes
0answers
26 views

What does a high cosine similarity score mean here?

I have a set of 50,000 documents divided into two classes. Class 1 has 5000 documents and, Class 2 has 45,000 documents Using word2vec embeddings, I extract 300 dimensional dense, real-valued ...
0
votes
1answer
35 views

Classification Model - How to Preprocess Text

I have a Dataframe that contains 2 columns: 'Skills' column - each cell contains a list of strings describing different technical and soft skills of a person, e.g: [Python,SQL,Java,Team Management,...
0
votes
0answers
23 views

Hierarchical Text classification

I am working on a project with a huge number of big groups of sentences! I need to classify each sentence based on the sentence and other sentences on the group. Actually there is 2 level of ...
0
votes
1answer
19 views

Text classification suggestion

I have a big data-set of sentences (tens of thousands) which some of these sentences are big but some are short. The main problem is that you should classify some sentences according to previous or ...
0
votes
0answers
12 views

How do I use additional features when doing sentiment analysis?

Say I have a dataset of 3 columns: text, topic and sentiment. Each row of the ...
0
votes
1answer
49 views

Hoes does laplace smoothing in Naive Bayes control high bias and high variance?

I'm trying to understand how laplace smoothing exactly helps to balance between overfitting and underfitting. I know that Laplace smoothing is used as a fail safe probability if there's a any ...
0
votes
0answers
65 views

TF-IDF versus TF for Cosine Similarity

My task is to examine emails and determine the cosine similarity between pairs of emails to see which ones are the same or almost the same. I was thinking of using the TF-IDF technique, but am not ...
0
votes
0answers
21 views

binary classification problem for data where one class potentially includes members of the other class

Situation Assuming one wants to classify text data into two sentiment categories: negative and neutral (i.e everything that is not negative). For the training process, labelled data is available. ...
0
votes
0answers
23 views

Classify documents using a set of known vocabularies

I have a bunch of documents that I want to classify which ones talk about soccer (unsupervised learning, I do not want to manually label the documents). One way I am thinking about is to go online ...
0
votes
0answers
29 views

Improve the accuracy of semantic text matching

I have a corpus of ~200K sentences of variable length, the median length is 16 words. My goal is for a given sentence to find other sentences with a similar meaning. I tried several approaches: using ...
0
votes
1answer
13 views

Should I convert classification output to integer and how?

I'm using a neural network to classify text, and the label of the training data is 0 or 1(i.e. binary classification). It works well in the training and evaluating process, but the prediction output ...
0
votes
0answers
13 views

classification association rules and closed frequent itemsets

if I need to mine all frequent itemsets for text classification using classification association rules. what is the most efficient algorithm? is CHARM algorithm efficient with mining only closed ...
0
votes
0answers
54 views

Visualizing a Latent Dirichlet Allocation (LDA) by Multidimensional Scaling (MDS)

I did an LDA with four topics for four different Smartphones. This was done using customer Reviews of Amazon. ...
0
votes
0answers
5 views

Combine Noisy predictions from an optical character recognition program

I am trying to perform optical character recognition on a field of text from different angles as the camera passes over it and beyond it. Due to the 3D skewing of the image, thee readings of the OCR ...
0
votes
1answer
25 views

Get association of categorical variables

I do a text analysis where I want to identify dependencies among categorical variables, for example let's take this dataset: ...
0
votes
0answers
44 views

Using LDA to create two layered wordcloud

I create a word-cloud using LDA model what I want to do is to find the documents IDs related to that topic group. So, for example, the image here I want to allow users to click the word broccoli and ...
0
votes
0answers
14 views

Correlations in text analysis - for an absolute beginner

I am analysing the media releases and speeches of two political leaders. I want to find out: how often they mention young people, and in what context. I know my frequency tables are correct because I ...
0
votes
0answers
28 views

How PV-DBOW works

The authors of the Paragraph Vector paper describe PV-DBOW with: 2.3. Paragraph Vector without word ordering: Distributed bag of words The above method considers the concatenation of the ...
0
votes
1answer
117 views

How to find correlation between text data?

I have data set similar to this: I want to if the columns subtype and item are correlated. They have different text, ...
0
votes
0answers
14 views

How to solve the estimate population characters with sample text-mining?

I've recently learning text-mining, but none of my textbooks talk of inferential statistics; they talk about how to analyze collected data but hardly deal with how to estimate population data through ...
0
votes
0answers
7 views

wordfish - must reference texts be at the absolute extreme of the spectrum

I have a question pertaining to the wordfish method (see here for the academic paper introducing the package and here for the package itself). My question regards ...
0
votes
0answers
23 views

How to statistically infer common pattern in text

Am trying to solve a problem where I need to infer common patterns in text for example, the data below, with bare eyes it can be noticed there is a pattern and that is ...
0
votes
0answers
11 views

Calculating the ratio of the (frequency of a specific word in the corpus/sum of the frequency of all the words)

I have the following code which gives me the list of top 10 words in a corpus in descending order of frequency: library(tidytext) tidybooks.nstop<-tidy_books %>% + anti_join(stop_words) ...
0
votes
0answers
21 views

Classification of sample with only unseen words

I'm doing text classification (Product Name) where one example belongs to one class. "Some Product Name" -> MODEL -> {CLASS_1 | CLASS_2 | CLASS_3 | CLASS_4} ...
2
votes
0answers
89 views

How to compute gain statistic for the multinomial Naive Bayes classifier from Jurafsky and Martin (2018)

I'm trying to figure out how to compute the gain statistic G(w) following the fitting of the multinomial Naive Bayes model. This statistic is described on p17 of the new edition of Jurafsky and ...
0
votes
0answers
29 views

Sentiment Analysis Issues Using R

I am using the R package sentimentr and sentiment function to get polarity/sentiment scores on a list of comments. The issue I am having is that the comments are in ...
0
votes
0answers
36 views

Difference between Semantic analysis and Syntactic analysis in text analysis

Can someone explain me clearly the meaning of Semantic analysis and Syntactic Analysis in text analysis ? I am very much confused between these two. Also tell the difference in how they are used in ...
0
votes
0answers
23 views

How to ensemble predictions from image classifier and text classifier?

I am doing multiclass classification based on images and text. I have predictions from both image classification and text. I am not sure how to combine them. Should I use probabilities as a feature to ...
0
votes
0answers
57 views

Guided LDA to categorize software requirements

I'm developing a application to categorize requirements in a requirement specification in to categories like database, front end, back end, etc. So for that I'm trying to use Guided LDA since labelled ...
0
votes
0answers
29 views

Pointwise Mutual Information Word Dependency

I have pre-defined concepts which are either a single word or couple of words that refer to a concept.( In the context of machine learning for instance, covariance matrix is a concept). I am trying to ...
2
votes
0answers
15 views

Correlation of phrases and answers to them

If I ask question in wrong forum, let me know, I'll delete it. I try find methods, methodology to create predicative model. Namely, I want to investigate the relationship between the sentences and ...
0
votes
0answers
17 views

naive bayes text classification: can I look at the individual word probabilities?

Assume I use the Naive Bayes classification algorithm. My question is simple: can I rank the words according to their posterior probabilities? I want to have a measure of "importance" of the words in ...
1
vote
1answer
13 views

Good way to use word similarity as a feature in supervised ML on text

I have a pretty low N data set of small sentences tagged with a label. I would like to create a classifier on this dataset. The word choice is not very variable since the domain is pretty specific. ...
0
votes
0answers
29 views

Using a priori knowledge in a classification task

I'm working on a classification task, related with text classification, where texts to be classified are requests for technical support, and the classes are technical guys which issues can be assigned ...
1
vote
1answer
198 views

Finding similar text - algorithms and evaluation

I've been asked to create a program that will rank similar texts to an input text given a collection of text. So far I've been using a tdidf representation and cosine similarity with a lot of regex-...
0
votes
0answers
64 views

Getting the document-per-topic loading using TextmineR package by passing term co-occurrence matrix

I am using TextmineR package to find the most similar documents to given document list. I used the following code to generate the tcm not dtm ...
0
votes
1answer
25 views

apply CountVectorized to whole data before applying train_test_split

Is there any difference between the two different snips of codes. ...
0
votes
0answers
9 views

Named Entity Recognizer for nominal phrases: Airbus vs Europe's biggest airplane manufacturer

i'm looking for a Named Entity Recognizer that is able to recognize "true names" as well as phrases that are synonymous for them. If possible i would like to use python. Can NLTK or spaCy models be ...
0
votes
0answers
51 views

Required sample size for sentiment analysis in customer satisfaction

Suppose we are interested in seeing how companies A through F are being viewed by different customers, and how this changes based on the customers' positions. A sample of customers is drawn from ...
1
vote
1answer
21 views

Information theoretic alternative to tf-idf heuristic?

I've been recently working with feature construction from texts, where tf-idf measure is one of the main options for vectorizing the documents (one feature per e.g., word). I was wondering, whether ...
0
votes
0answers
93 views

How to encode text and categorical variables together?

I have two groups of texts that are very similar (e.g. reviews written on fridays and reviews written on mondays), and I want to build a LSTM that can classify them into positive and negative reviews. ...
0
votes
0answers
25 views

What is the best model for keyphrase extraction from super long text?

I’m working on a keyphrase extraction task. The biggest difficulty of this task is that the text is very long (5000-20000 words). I’ve tried several unsupervised algorithms such as Tf-idf and TextRank ...
1
vote
1answer
711 views

Imbalanced multiclass classification with many classes

I am working on a text classification project in which we have hundreds of (imbalanced) classes. Some characteristics of the data: We have examples of "bad" documents. Basically documents that don't ...
0
votes
0answers
98 views

Applying Label From Supervised Learning to Unlabeled Data- Text Classification

I am wondering if anyone has code to following: 1) Apply labels from a previous text classification dataset like this type of data (https://colab.research.google.com/drive/...
0
votes
1answer
72 views

text preprocessing using keras [closed]

I am getting started with NLP, in kaggle , and it dont get how this keras preprocessing works if anyone could explain the code would be much helpful,thanks ...