Questions tagged [text-mining]
Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.
640
questions
0
votes
0
answers
14
views
Would it be wise to feed k-means results into a cmeans?
I am conducting a cluster analysis of documents using document embeddings as the input to the algorithms. One of the problems I am coming across is that in reality there are documents that belong to ...
- 31
0
votes
0
answers
10
views
How does Doc2Vec handle documents where the length is less than the window size?
I am using doc2vec to vectorise documents that average in length 10 words (PV-DBOW implementation of the algorithm). I am wondering how doc2vec handles cases where the number of words in a document is ...
- 31
0
votes
1
answer
51
views
Using Word Embeddings in Clustering and Topic Modelling
I am new to the field of NLP and would appreciate any guidance please. I am trying to understand how word embeddings can be used in clustering and topic modelling. If I create word embeddings for ...
- 31
0
votes
0
answers
16
views
Why are my deep learning models giving unreasonably high accuracy on test data?
I'm trying to do sarcasm detection on Twitter data to replicate the results mentioned in this paper. Binary classification problem. For that I used a separate set of unlabeled tweets to create the ...
- 119
0
votes
0
answers
27
views
Adjusted TF-IDF where many terms appear in every document
Struggling with something so hoped the brilliant minds of the internet could help me out.
I have a large dataset of job postings from which I have extracted the skill demand (no. of times a skill is ...
- 3
0
votes
0
answers
8
views
Text Classification model unable to learn [duplicate]
I am trying to build a text classification model. When I train the model it is unable to improve accuracy and at some point accuracy even decreases and loss increases.
I have researched for possible ...
- 1
1
vote
0
answers
46
views
Similiarity between two corpus of text
I have two separated corpus of text, and i would like to understand wheter these are similiar or not using cosine similarity.
I'm not sure on how to approach this problem, but i was thinking as a ...
- 33
0
votes
0
answers
48
views
Transformer-based model for semantic linking
I am working on a Form Understanding project, I have applied OCR and LayoutLM to get the BIEOS labels for all text(e.g. question, answers, others etc.), but I found layoutLM does not provide the ...
- 313
1
vote
1
answer
213
views
How to avoid underflow of the probability of sentence in calculating the perplexity of corpus
I am looking at this post How to find the perplexity of a corpus. I understand the whole post, but
the probability of a sentence appear in a corpus, in a unigram model,
is given by p(s)=∏ni=1p(wi), ...
- 13
1
vote
0
answers
16
views
processing natural language that descripe time frequency with R
I'm dealing with data that descripe onset frequency of a symptom. The text in each cell was not in the same format. For example:
...
- 73
1
vote
0
answers
33
views
Does average or max pooling actually summarise the sentence?
I am working on an multi-label text classification problem at work and adapted model architecture from this notebook of Toxic Comment Classification challenge on Kaggle.
I have trained the model, a ...
2
votes
1
answer
51
views
Text similarity for badly written text
Consider the following scenario:
Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
- 287
1
vote
1
answer
22
views
How to extract numerical features that can separate well documents belonging to two different classes?
I have a group of texts belonging to two different classes. I would like to extract numerical features that can separate well the two classes.
Right now I implemented a classic TF-IDF with a document ...
- 21
2
votes
0
answers
33
views
ML generated word choice to create distinct "speakers" [closed]
How hard a project would it be to use ML to assist a single author/script writer in writing dialog where each "speaker" sounds like a distinct person?
Is that something that a professional ...
- 121
1
vote
3
answers
320
views
How to improve language model ex: BERT on unseen text in training?
so I am using pre-trained language model for binary classification. I fine-tune the model by training on data my downstream task. The results are good almost 98% F-measure.
However, when I remove a ...
- 11
1
vote
0
answers
43
views
How to statistically compare the frequencies of two different words in a single corpus
Suppose I have a large corpus of text data and I would like to compare the frequencies of words $w_1$ and $w_2$. How would I go about testing whether or not their respective frequencies, $f_1$ and $...
- 11
0
votes
1
answer
69
views
Search, rank and recommend in large text datasets
Imagine you are Spotify and you have billions of songs. Assume that each of these songs are transcribed into text. How do you design your search and recommendation pipeline such that when somebody ...
- 148
0
votes
1
answer
32
views
How to extract FSAs from postal codes when there is no match?
I would like to extract Canadian FSAs from unstrucured data. I want to pull only the first instance of each match.
The problem: Some data don't include postal code and my function won't produce the ...
0
votes
1
answer
224
views
How to compare two texts with different order of words?
I have two texts, one ground truth and one OCR result, and I want to measure to what accuracy the result matches the ground truth. But since the text source is non-linear, both texts have a different ...
- 1
1
vote
0
answers
34
views
Calculating the quality of text parsing script
I have a python script that picks relevant sentences from a text corpus based on keywords and stopwords and applies some classifiers for the chosen sentences. The context is academic research.
The ...
- 11
1
vote
0
answers
22
views
Unable to understand this Bag of words implementation [closed]
I am working on a chatbot project
Input Text format
List of the list in below sequence for multiple categories.
[text from user, category of text, reply by chatbot]
...
- 111
0
votes
1
answer
127
views
How do you use a TF-IDF matrix to score text similarity?
I'm trying to match strings of words with a website which has bulletpoints from all of the URL's I'm interested in whose text is most similar to it. The way I thought of doing it is to get all of the ...
- 121
1
vote
0
answers
23
views
How would I create a machine learning based text parser? [closed]
I have many documents that have the same sections, but have different formatting. How could I make a model to segment the document into given sections?
I have tried making classifiers for specific ...
- 31
0
votes
0
answers
114
views
Machine Learning Method to Predict Continuous Target Variable with Text Data Using R
Purpose
I have a surgery dataset with surgery id (i.e.,oid), the procedure names (o3.name.procedure), and the actual duration (i....
- 131
0
votes
0
answers
50
views
What is a good method to compare the values with different sample sizes?
I am performing text analysis on some social media posts and I have a list of words and corresponding engagement rates, such as the list below (a much shorter version of a long list). As you can see, ...
3
votes
1
answer
499
views
How are sentences one-hot encoded internally in an Embedding Keras Layer?
Multiple references are clear on how a single word is one-hot encoded in an Embedding layer, but what about sentences?
In order to illustrate an example, I will use the following SO reference. Let's ...
2
votes
0
answers
233
views
Question related to using Pooled Output from BERT for similarity between sentences
I was hoping someone could give me advice and feedback on my current approach and possibly suggest to me a possible alternative. I'm trying to find the sentences that are most similar using the pooled ...
- 21
1
vote
0
answers
324
views
How does the pooled output from the output layer in a BERT model reference back to the actual text?
I was wondering if someone can refer to me a source or describe to me how to interpret the 768 sequence of numbers that are derived from the output layer of the BERT Model. Like, what do they mean and ...
- 167
0
votes
0
answers
57
views
How to perform text classification on unlabelled test data?
I am using TF-IDF to perform feature extraction and then passing the sparse matrix to perform training along with text data which is also transformed to sparse matrix. I understand that the input to ...
- 79
1
vote
1
answer
327
views
Latent Dirichlet Allocation and topic distributions
When reading about the LDA, the generative procedure for a document is often presented as follows:
For each topic $k\in \{1,\ldots, K\}$
Draw a distribution over words $\phi_k\sim \text{Dirichlet}(\...
- 165
3
votes
1
answer
834
views
Best practice for named entity recognition on large texts
What are the best practices to apply NER to large texts (e.g 20 pages+)?
One common advice is to split the text before passing it as input to the model. However this can require a significant manual ...
- 502
3
votes
1
answer
523
views
Clustering after t-SNE in R
As explained here, t-SNE maps high dimensional data such as word embedding into a lower dimension in such that the distance between two words roughly describe the similarity. It also begins to create ...
- 297
1
vote
1
answer
155
views
How many emails would I need to train a good text extraction model?
I'm looking to train a model that will identify product names in an email that a user has bought. The end result would be something very much like named entity extraction, except this should correctly ...
- 13
1
vote
0
answers
40
views
Correlation of a matrix vs Conditional probability of a matrix
I am confused how to justify that why eigenvalues of correlation matrix of a document-term matrix (dtm) is different from the eigenvalues of matrix of conditional probability between the terms?
In ...
- 11
0
votes
0
answers
28
views
Clustering mixed data based on text anlysis: Approach evaluation
As part of my project, I've been trying to analyse (and hopefully make some knowledgeable conclusions about) the movie database dataset, which consists of the following columns:
Movie ID - ID of a ...
0
votes
0
answers
48
views
Clustering mixed data based on text anlysis: Sparse Matrix problem
Good day/evening/any other time of the day!
As part of my project, I've been trying to analyse (and hopefully make some konwlegable conclusions about) the movie database dataset, which consists of the ...
1
vote
0
answers
120
views
When would you use purity as a measure of external validity over entropy? [closed]
This question particularly pertains to text clustering. I've not really found anything on why one would use purity over entropy or vice versa. Could someone explain this to me?
- 167
5
votes
1
answer
784
views
How to use LDA to classify documents into pre defined topics
LDA is unsupervised and it classifies documents into topics. But, is there a way to make the LDA classify the documents into the predefined (or specific desired) topics.
Below link says we need custom ...
- 787
1
vote
0
answers
24
views
Test if semantic triple occurs more often than by chance
I have a large table (more than 9,000,000 rows) of semantic predications (i.e., triples of subject-predicate-object) extracted from sentences of scientific abstracts. The data are organized in the ...
- 2,203
1
vote
0
answers
34
views
Question text Mining using Random Forest and PCA [closed]
I'm currently using the Reuters 50-50 dataset (https://archive.ics.uci.edu/ml/datasets/Reuter_50_50) to predict authorship. I've tried to first use PCA on both the test and training dataset to get a ...
- 167
0
votes
1
answer
25
views
Extracting information from form document through supervised learning
I was searching for a while around the web and I couldn't find any solution that would give some ideas on how to solve my problem. I have a few hundreds of document with some permission forms filled ...
- 101
1
vote
0
answers
26
views
Multilabel Tweet Classification
I need some general advice and possible ideas.
Problem statement goes like this --
We are given a tweet and we have to specify associated labels for it like generalized hate, support, oppose, ...
- 121
1
vote
0
answers
49
views
Significance of datapoint outside prediction band
With linear regression I am plotting 25 bodies of text with their vocabulary count (independent variable X) and occurrence of a particular word (for example: "this").
I have a linear ...
- 11
0
votes
1
answer
110
views
Feature Selection in Twitter Sentiment Analysis
I'm currently working on a twitter sentiment analysis project. In this project, one requirement is to perform Feature selection for a better prediction. But I'm fairly confused about the techniques to ...
- 31
0
votes
1
answer
83
views
Text composition based on categorical features
The problem I have to solve is to find a model that links categorical features (bool type actually) to text documents.
The categorical features are answers to questions. Any different combination of ...
1
vote
1
answer
442
views
Combining two sequences for text classification
I'm doing text classification on comments posted on articles/stories. The two human-labeled classes are appropriate and not appropriate (not the same as happy/angry or any "sentiment" ...
- 1,238
0
votes
1
answer
129
views
Are the vectorization settings considered hyperparameters in ML?
Short definition of HP:
"In machine learning, a hyperparameter is a parameter whose value is
set before the learning process begins. Hyperparameter optimization or
tuning is the problem of ...
1
vote
1
answer
684
views
Using POS Tags and NERs as Features for Text Classification or Sentiment Analysis
I am trying to implement text classification and sentiment analysis from the documents.
I always use POS tags as features in the following way.
Mike is playing football
I would convert it into ...
- 31
1
vote
0
answers
54
views
Multi-label Text class
The data i am dealing with are simple text sentences that needs to be classified into variaous labels that correspond to the different topics as simple as Yes/No class. Several labels can be assigned ...
- 11
0
votes
0
answers
31
views
Real life class imbalance [duplicate]
Fellow like-minded people,
I'm writing my thesis in fake news detection on scrapped twitter data and facing an issue (among many others). Fake news consist of less than 10% of the total tweets or ...