Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
1 vote
1 answer
25 views

Text similarity for badly written text

Consider the following scenario: Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
user avatar
0 votes
0 answers
22 views

Text classification of posts with very small dataset

I hope this is the right place to ask this. I am a student working on a project about the prediction of the privacy policy applied to textual posts on facebook. The objective is to predict for a post ...
user avatar
0 votes
0 answers
16 views

How to design a method for finding multiword terms based on labeled data

Setup: I have many textdocuments that have been processed by an OCR Engine. These documents are Invoices and the endgoal is to classify words inside each document. If words on a document are seperated ...
user avatar
1 vote
1 answer
16 views

How to extract numerical features that can separate well documents belonging to two different classes?

I have a group of texts belonging to two different classes. I would like to extract numerical features that can separate well the two classes. Right now I implemented a classic TF-IDF with a document ...
user avatar
0 votes
0 answers
16 views

Text analysis how to handle very similar n-grams

I have stemmed and tf-idf'd some text for use with LDA topic extraction. My task is to examine some of the top n-gram > 2 results. Initially, I had presumed that ...
user avatar
2 votes
0 answers
31 views

ML generated word choice to create distinct "speakers" [closed]

How hard a project would it be to use ML to assist a single author/script writer in writing dialog where each "speaker" sounds like a distinct person? Is that something that a professional ...
user avatar
  • 121
1 vote
3 answers
60 views

How to improve language model ex: BERT on unseen text in training?

so I am using pre-trained language model for binary classification. I fine-tune the model by training on data my downstream task. The results are good almost 98% F-measure. However, when I remove a ...
user avatar
1 vote
0 answers
29 views

How to statistically compare the frequencies of two different words in a single corpus

Suppose I have a large corpus of text data and I would like to compare the frequencies of words $w_1$ and $w_2$. How would I go about testing whether or not their respective frequencies, $f_1$ and $...
user avatar
  • 11
0 votes
1 answer
39 views

Search, rank and recommend in large text datasets

Imagine you are Spotify and you have billions of songs. Assume that each of these songs are transcribed into text. How do you design your search and recommendation pipeline such that when somebody ...
user avatar
  • 118
0 votes
0 answers
49 views

How to fix label errors in text corpus?

I have a huge text corpus (around 60k+ documents with 40 classes. But the corpus suffers from class imbalance problem. Also, the ...
user avatar
  • 185
0 votes
1 answer
24 views

How to extract FSAs from postal codes when there is no match?

I would like to extract Canadian FSAs from unstrucured data. I want to pull only the first instance of each match. The problem: Some data don't include postal code and my function won't produce the ...
user avatar
0 votes
1 answer
58 views

How to compare two texts with different order of words?

I have two texts, one ground truth and one OCR result, and I want to measure to what accuracy the result matches the ground truth. But since the text source is non-linear, both texts have a different ...
user avatar
1 vote
0 answers
21 views

Calculating the quality of text parsing script

I have a python script that picks relevant sentences from a text corpus based on keywords and stopwords and applies some classifiers for the chosen sentences. The context is academic research. The ...
user avatar
  • 11
1 vote
0 answers
21 views

Unable to understand this Bag of words implementation [closed]

I am working on a chatbot project Input Text format List of the list in below sequence for multiple categories. [text from user, category of text, reply by chatbot] ...
user avatar
  • 11
0 votes
0 answers
10 views

document subsimilarity matching

I'm looking to classify subsections of "full" documents based on their similarity to a set of subsections that have been manually curated and assigned labels (let's call these short ...
user avatar
  • 101
0 votes
1 answer
28 views

How do you use a TF-IDF matrix to score text similarity?

I'm trying to match strings of words with a website which has bulletpoints from all of the URL's I'm interested in whose text is most similar to it. The way I thought of doing it is to get all of the ...
user avatar
  • 121
1 vote
0 answers
8 views

How would I create a machine learning based text parser? [closed]

I have many documents that have the same sections, but have different formatting. How could I make a model to segment the document into given sections? I have tried making classifiers for specific ...
user avatar
  • 21
0 votes
0 answers
44 views

Machine Learning Method to Predict Continuous Target Variable with Text Data Using R

Purpose I have a surgery dataset with surgery id (i.e.,oid), the procedure names (o3.name.procedure), and the actual duration (i....
user avatar
  • 11
0 votes
0 answers
15 views

Can we find the semantic similarity between two words in same document in encoded matrix from Latent Semantic Analysis?

Question The following the is encoded matrix obtained through Latent Semantic Analysis, my final goal is to find if we can get the semantic similarity between two words in one topic let's say topic 1 ...
user avatar
1 vote
0 answers
55 views

Alternative to tf-idf for 2 documents?

I am currently trying to construct word clouds between two tidy text documents (tweets). My question is methodological, although I am using bind_tf_idf in R for the ...
user avatar
0 votes
0 answers
45 views

What is a good method to compare the values with different sample sizes?

I am performing text analysis on some social media posts and I have a list of words and corresponding engagement rates, such as the list below (a much shorter version of a long list). As you can see, ...
user avatar
0 votes
0 answers
26 views

eBay search phrase to listings relevance

Working on a quick PoC, and it’s based on eBay data. Basically, someone puts in a search phrase and receives a number of listings. These listings have been classified (not by eBay, but by my colleague)...
user avatar
3 votes
1 answer
132 views

How are sentences one-hot encoded internally in an Embedding Keras Layer?

Multiple references are clear on how a single word is one-hot encoded in an Embedding layer, but what about sentences? In order to illustrate an example, I will use the following SO reference. Let's ...
user avatar
0 votes
0 answers
29 views

Can I use the masi distance to compare coders where the text document is not exactly the same?

It seems that Passoneau's MASI distance https://academiccommons.columbia.edu/doi/10.7916/D8PV6TTT was meant to grade different judges on the same document. Could I use this distance method to compare ...
user avatar
  • 1
0 votes
0 answers
38 views

Should bag of words in training set include test set data when doing text classification?

I'm doing text classification to identify 'attacks' from wikipedia comments using a simple bag of words model and a linear SVM classifier. Because of class imbalance I'm using F1 score as my error ...
user avatar
  • 1
2 votes
0 answers
95 views

Question related to using Pooled Output from BERT for similarity between sentences

I was hoping someone could give me advice and feedback on my current approach and possibly suggest to me a possible alternative. I'm trying to find the sentences that are most similar using the pooled ...
user avatar
  • 21
1 vote
0 answers
237 views

How does the pooled output from the output layer in a BERT model reference back to the actual text?

I was wondering if someone can refer to me a source or describe to me how to interpret the 768 sequence of numbers that are derived from the output layer of the BERT Model. Like, what do they mean and ...
user avatar
0 votes
0 answers
29 views

How to perform text classification on unlabelled test data?

I am using TF-IDF to perform feature extraction and then passing the sparse matrix to perform training along with text data which is also transformed to sparse matrix. I understand that the input to ...
user avatar
  • 69
1 vote
1 answer
164 views

Latent Dirichlet Allocation and topic distributions

When reading about the LDA, the generative procedure for a document is often presented as follows: For each topic $k\in \{1,\ldots, K\}$ Draw a distribution over words $\phi_k\sim \text{Dirichlet}(\...
user avatar
  • 145
2 votes
1 answer
269 views

Best practice for named entity recognition on large texts

What are the best practices to apply NER to large texts (e.g 20 pages+)? One common advice is to split the text before passing it as input to the model. However this can require a significant manual ...
user avatar
  • 472
1 vote
1 answer
299 views

Clustering after t-SNE in R

As explained here, t-SNE maps high dimensional data such as word embedding into a lower dimension in such that the distance between two words roughly describe the similarity. It also begins to create ...
user avatar
  • 187
1 vote
1 answer
20 views

How many emails would I need to train a good text extraction model?

I'm looking to train a model that will identify product names in an email that a user has bought. The end result would be something very much like named entity extraction, except this should correctly ...
user avatar
1 vote
0 answers
32 views

Correlation of a matrix vs Conditional probability of a matrix

I am confused how to justify that why eigenvalues of correlation matrix of a document-term matrix (dtm) is different from the eigenvalues of matrix of conditional probability between the terms? In ...
user avatar
  • 11
0 votes
0 answers
21 views

Clustering mixed data based on text anlysis: Approach evaluation

As part of my project, I've been trying to analyse (and hopefully make some knowledgeable conclusions about) the movie database dataset, which consists of the following columns: Movie ID - ID of a ...
user avatar
0 votes
0 answers
25 views

Clustering mixed data based on text anlysis: Sparse Matrix problem

Good day/evening/any other time of the day! As part of my project, I've been trying to analyse (and hopefully make some konwlegable conclusions about) the movie database dataset, which consists of the ...
user avatar
1 vote
0 answers
43 views

When would you use purity as a measure of external validity over entropy? [closed]

This question particularly pertains to text clustering. I've not really found anything on why one would use purity over entropy or vice versa. Could someone explain this to me?
user avatar
5 votes
1 answer
407 views

How to use LDA to classify documents into pre defined topics

LDA is unsupervised and it classifies documents into topics. But, is there a way to make the LDA classify the documents into the predefined (or specific desired) topics. Below link says we need custom ...
user avatar
  • 717
1 vote
0 answers
16 views

Test if semantic triple occurs more often than by chance

I have a large table (more than 9,000,000 rows) of semantic predications (i.e., triples of subject-predicate-object) extracted from sentences of scientific abstracts. The data are organized in the ...
user avatar
  • 2,141
1 vote
0 answers
19 views

Question text Mining using Random Forest and PCA [closed]

I'm currently using the Reuters 50-50 dataset (https://archive.ics.uci.edu/ml/datasets/Reuter_50_50) to predict authorship. I've tried to first use PCA on both the test and training dataset to get a ...
user avatar
0 votes
1 answer
20 views

Extracting information from form document through supervised learning

I was searching for a while around the web and I couldn't find any solution that would give some ideas on how to solve my problem. I have a few hundreds of document with some permission forms filled ...
user avatar
  • 101
1 vote
0 answers
19 views

Multilabel Tweet Classification

I need some general advice and possible ideas. Problem statement goes like this -- We are given a tweet and we have to specify associated labels for it like generalized hate, support, oppose, ...
user avatar
  • 121
1 vote
0 answers
29 views

Significance of datapoint outside prediction band

With linear regression I am plotting 25 bodies of text with their vocabulary count (independent variable X) and occurrence of a particular word (for example: "this"). I have a linear ...
user avatar
  • 11
0 votes
1 answer
52 views

Feature Selection in Twitter Sentiment Analysis

I'm currently working on a twitter sentiment analysis project. In this project, one requirement is to perform Feature selection for a better prediction. But I'm fairly confused about the techniques to ...
user avatar
0 votes
1 answer
48 views

Text composition based on categorical features

The problem I have to solve is to find a model that links categorical features (bool type actually) to text documents. The categorical features are answers to questions. Any different combination of ...
user avatar
1 vote
1 answer
247 views

Combining two sequences for text classification

I'm doing text classification on comments posted on articles/stories. The two human-labeled classes are appropriate and not appropriate (not the same as happy/angry or any "sentiment" ...
user avatar
0 votes
1 answer
66 views

Are the vectorization settings considered hyperparameters in ML?

Short definition of HP: "In machine learning, a hyperparameter is a parameter whose value is set before the learning process begins. Hyperparameter optimization or tuning is the problem of ...
user avatar
1 vote
1 answer
444 views

Using POS Tags and NERs as Features for Text Classification or Sentiment Analysis

I am trying to implement text classification and sentiment analysis from the documents. I always use POS tags as features in the following way. Mike is playing football I would convert it into ...
user avatar
1 vote
0 answers
29 views

Multi-label Text class

The data i am dealing with are simple text sentences that needs to be classified into variaous labels that correspond to the different topics as simple as Yes/No class. Several labels can be assigned ...
user avatar
  • 11
0 votes
0 answers
27 views

Real life class imbalance [duplicate]

Fellow like-minded people, I'm writing my thesis in fake news detection on scrapped twitter data and facing an issue (among many others). Fake news consist of less than 10% of the total tweets or ...
user avatar
2 votes
1 answer
1k views

ROUGE scores for extractive vs abstractive text summarization

The ROUGE score (scores) allows us to measure (although not in a perfect way) the quality of our text summarization by computing the frequency of overlapping n-grams between our produced summary and ...
user avatar
  • 123

1
2 3 4 5
13