Skip to main content

Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
0 votes
0 answers
15 views

One class text classification for open-ended survey questions

I have a specific text analysis problem, and hope someone can point me in the direction of a good approach. I have three corpuses consisting of short documents (open-ended survey questions). One is a &...
soran hajo dahl's user avatar
0 votes
0 answers
8 views

Problems in understanding Word2vec architectures

I have probably a very simple question, but I did not find any clear resource on the web. First let's consider the Skip-gram model, in which we try to predict a context word given the target word. In ...
user405969's user avatar
0 votes
0 answers
14 views

Classify text by topics using SVM: Derive the upper bound for the norm of the weight vector

In the section on SVM in one book I'm reading, the authors wrote: Consider the problem of learning to classify a short text document according to its topic, say, whether the document is about sports ...
Tran Khanh's user avatar
0 votes
0 answers
9 views

Using Unordered Text Features in Machine Learning Model

I am building a classification model using deidentified patient data with ICD-10 codes as inputs. Each code is a string and represents a diagnosis, and these follow the pattern of 1 letter, followed ...
Omnitragedy's user avatar
0 votes
0 answers
21 views

Best way to classify text based on a set of seed/keywords

This question was asked six years ago but figured it was worth an update. I'm attempting to filter news data based on a set of keywords. I think the ideal process for this would be some form of: ...
creekjumper's user avatar
0 votes
0 answers
14 views

Would it be wise to feed k-means results into a cmeans?

I am conducting a cluster analysis of documents using document embeddings as the input to the algorithms. One of the problems I am coming across is that in reality there are documents that belong to ...
osckt's user avatar
  • 31
0 votes
0 answers
28 views

How does Doc2Vec handle documents where the length is less than the window size?

I am using doc2vec to vectorise documents that average in length 10 words (PV-DBOW implementation of the algorithm). I am wondering how doc2vec handles cases where the number of words in a document is ...
osckt's user avatar
  • 31
0 votes
1 answer
465 views

Using Word Embeddings in Clustering and Topic Modelling

I am new to the field of NLP and would appreciate any guidance please. I am trying to understand how word embeddings can be used in clustering and topic modelling. If I create word embeddings for ...
osckt's user avatar
  • 31
0 votes
0 answers
37 views

Adjusted TF-IDF where many terms appear in every document

Struggling with something so hoped the brilliant minds of the internet could help me out. I have a large dataset of job postings from which I have extracted the skill demand (no. of times a skill is ...
Dandae's user avatar
  • 3
1 vote
0 answers
51 views

Similiarity between two corpus of text

I have two separated corpus of text, and i would like to understand wheter these are similiar or not using cosine similarity. I'm not sure on how to approach this problem, but i was thinking as a ...
user373562's user avatar
1 vote
1 answer
380 views

How to avoid underflow of the probability of sentence in calculating the perplexity of corpus

I am looking at this post How to find the perplexity of a corpus. I understand the whole post, but the probability of a sentence appear in a corpus, in a unigram model, is given by p(s)=∏ni=1p(wi), ...
Qqqq's user avatar
  • 13
1 vote
0 answers
16 views

processing natural language that descripe time frequency with R

I'm dealing with data that descripe onset frequency of a symptom. The text in each cell was not in the same format. For example: ...
Ian Wang's user avatar
1 vote
0 answers
37 views

Does average or max pooling actually summarise the sentence?

I am working on an multi-label text classification problem at work and adapted model architecture from this notebook of Toxic Comment Classification challenge on Kaggle. I have trained the model, a ...
Naveen Reddy Marthala's user avatar
2 votes
1 answer
57 views

Text similarity for badly written text

Consider the following scenario: Suppose two lists of words $L_{1}$ and $L_{2}$ are given. $L_{1}$ contains just bad-written phrases (like 'age' instead of '4ge' or 'blwe' instead of 'blue' etc.). On ...
Ramiro Hum-Sah's user avatar
1 vote
1 answer
24 views

How to extract numerical features that can separate well documents belonging to two different classes?

I have a group of texts belonging to two different classes. I would like to extract numerical features that can separate well the two classes. Right now I implemented a classic TF-IDF with a document ...
inginging's user avatar
2 votes
0 answers
34 views

ML generated word choice to create distinct "speakers" [closed]

How hard a project would it be to use ML to assist a single author/script writer in writing dialog where each "speaker" sounds like a distinct person? Is that something that a professional ...
BCS's user avatar
  • 121
1 vote
3 answers
422 views

How to improve language model ex: BERT on unseen text in training?

so I am using pre-trained language model for binary classification. I fine-tune the model by training on data my downstream task. The results are good almost 98% F-measure. However, when I remove a ...
Injy Sarhan's user avatar
1 vote
0 answers
49 views

How to statistically compare the frequencies of two different words in a single corpus

Suppose I have a large corpus of text data and I would like to compare the frequencies of words $w_1$ and $w_2$. How would I go about testing whether or not their respective frequencies, $f_1$ and $...
Joshua's user avatar
  • 11
0 votes
1 answer
79 views

Search, rank and recommend in large text datasets

Imagine you are Spotify and you have billions of songs. Assume that each of these songs are transcribed into text. How do you design your search and recommendation pipeline such that when somebody ...
mhsnk's user avatar
  • 307
0 votes
1 answer
32 views

How to extract FSAs from postal codes when there is no match?

I would like to extract Canadian FSAs from unstrucured data. I want to pull only the first instance of each match. The problem: Some data don't include postal code and my function won't produce the ...
sometimes_r's user avatar
0 votes
1 answer
257 views

How to compare two texts with different order of words?

I have two texts, one ground truth and one OCR result, and I want to measure to what accuracy the result matches the ground truth. But since the text source is non-linear, both texts have a different ...
Shanksum's user avatar
1 vote
0 answers
35 views

Calculating the quality of text parsing script

I have a python script that picks relevant sentences from a text corpus based on keywords and stopwords and applies some classifiers for the chosen sentences. The context is academic research. The ...
Peter's user avatar
  • 11
1 vote
0 answers
22 views

Unable to understand this Bag of words implementation [closed]

I am working on a chatbot project Input Text format List of the list in below sequence for multiple categories. [text from user, category of text, reply by chatbot] ...
pquest's user avatar
  • 111
0 votes
1 answer
163 views

How do you use a TF-IDF matrix to score text similarity?

I'm trying to match strings of words with a website which has bulletpoints from all of the URL's I'm interested in whose text is most similar to it. The way I thought of doing it is to get all of the ...
sangstar's user avatar
  • 131
1 vote
0 answers
48 views

How would I create a machine learning based text parser? [closed]

I have many documents that have the same sections, but have different formatting. How could I make a model to segment the document into given sections? I have tried making classifiers for specific ...
Jason p's user avatar
  • 31
0 votes
0 answers
133 views

Machine Learning Method to Predict Continuous Target Variable with Text Data Using R

Purpose I have a surgery dataset with surgery id (i.e.,oid), the procedure names (o3.name.procedure), and the actual duration (i....
J.K.'s user avatar
  • 143
0 votes
0 answers
50 views

What is a good method to compare the values with different sample sizes?

I am performing text analysis on some social media posts and I have a list of words and corresponding engagement rates, such as the list below (a much shorter version of a long list). As you can see, ...
James Carpenter's user avatar
3 votes
1 answer
621 views

How are sentences one-hot encoded internally in an Embedding Keras Layer?

Multiple references are clear on how a single word is one-hot encoded in an Embedding layer, but what about sentences? In order to illustrate an example, I will use the following SO reference. Let's ...
Fernando Wittmann's user avatar
2 votes
0 answers
285 views

Question related to using Pooled Output from BERT for similarity between sentences

I was hoping someone could give me advice and feedback on my current approach and possibly suggest to me a possible alternative. I'm trying to find the sentences that are most similar using the pooled ...
marco's user avatar
  • 21
1 vote
0 answers
335 views

How does the pooled output from the output layer in a BERT model reference back to the actual text?

I was wondering if someone can refer to me a source or describe to me how to interpret the 768 sequence of numbers that are derived from the output layer of the BERT Model. Like, what do they mean and ...
Kliocontar's user avatar
0 votes
0 answers
66 views

How to perform text classification on unlabelled test data?

I am using TF-IDF to perform feature extraction and then passing the sparse matrix to perform training along with text data which is also transformed to sparse matrix. I understand that the input to ...
Anees's user avatar
  • 79
2 votes
1 answer
399 views

Latent Dirichlet Allocation and topic distributions

When reading about the LDA, the generative procedure for a document is often presented as follows: For each topic $k\in \{1,\ldots, K\}$ Draw a distribution over words $\phi_k\sim \text{Dirichlet}(\...
Tyler D's user avatar
  • 225
3 votes
1 answer
1k views

Best practice for named entity recognition on large texts

What are the best practices to apply NER to large texts (e.g 20 pages+)? One common advice is to split the text before passing it as input to the model. However this can require a significant manual ...
mobupu's user avatar
  • 532
3 votes
1 answer
586 views

Clustering after t-SNE in R

As explained here, t-SNE maps high dimensional data such as word embedding into a lower dimension in such that the distance between two words roughly describe the similarity. It also begins to create ...
Mark's user avatar
  • 307
1 vote
1 answer
278 views

How many emails would I need to train a good text extraction model?

I'm looking to train a model that will identify product names in an email that a user has bought. The end result would be something very much like named entity extraction, except this should correctly ...
user2578330's user avatar
1 vote
0 answers
42 views

Correlation of a matrix vs Conditional probability of a matrix

I am confused how to justify that why eigenvalues of correlation matrix of a document-term matrix (dtm) is different from the eigenvalues of matrix of conditional probability between the terms? In ...
Roze's user avatar
  • 11
0 votes
0 answers
28 views

Clustering mixed data based on text anlysis: Approach evaluation

As part of my project, I've been trying to analyse (and hopefully make some knowledgeable conclusions about) the movie database dataset, which consists of the following columns: Movie ID - ID of a ...
Andrzej Chomiczewski's user avatar
0 votes
0 answers
49 views

Clustering mixed data based on text anlysis: Sparse Matrix problem

Good day/evening/any other time of the day! As part of my project, I've been trying to analyse (and hopefully make some konwlegable conclusions about) the movie database dataset, which consists of the ...
Andrzej Chomiczewski's user avatar
1 vote
0 answers
151 views

When would you use purity as a measure of external validity over entropy? [closed]

This question particularly pertains to text clustering. I've not really found anything on why one would use purity over entropy or vice versa. Could someone explain this to me?
Kliocontar's user avatar
5 votes
1 answer
924 views

How to use LDA to classify documents into pre defined topics

LDA is unsupervised and it classifies documents into topics. But, is there a way to make the LDA classify the documents into the predefined (or specific desired) topics. Below link says we need custom ...
tjt's user avatar
  • 827
1 vote
0 answers
25 views

Test if semantic triple occurs more often than by chance

I have a large table (more than 9,000,000 rows) of semantic predications (i.e., triples of subject-predicate-object) extracted from sentences of scientific abstracts. The data are organized in the ...
Andrej's user avatar
  • 2,213
1 vote
0 answers
34 views

Question text Mining using Random Forest and PCA [closed]

I'm currently using the Reuters 50-50 dataset (https://archive.ics.uci.edu/ml/datasets/Reuter_50_50) to predict authorship. I've tried to first use PCA on both the test and training dataset to get a ...
Kliocontar's user avatar
0 votes
1 answer
31 views

Extracting information from form document through supervised learning

I was searching for a while around the web and I couldn't find any solution that would give some ideas on how to solve my problem. I have a few hundreds of document with some permission forms filled ...
Primoz's user avatar
  • 101
1 vote
0 answers
31 views

Multilabel Tweet Classification

I need some general advice and possible ideas. Problem statement goes like this -- We are given a tweet and we have to specify associated labels for it like generalized hate, support, oppose, ...
Vineet's user avatar
  • 121
1 vote
0 answers
50 views

Significance of datapoint outside prediction band

With linear regression I am plotting 25 bodies of text with their vocabulary count (independent variable X) and occurrence of a particular word (for example: "this"). I have a linear ...
Hakbijl's user avatar
  • 11
0 votes
1 answer
115 views

Feature Selection in Twitter Sentiment Analysis

I'm currently working on a twitter sentiment analysis project. In this project, one requirement is to perform Feature selection for a better prediction. But I'm fairly confused about the techniques to ...
Ruffybeo's user avatar
0 votes
1 answer
87 views

Text composition based on categorical features

The problem I have to solve is to find a model that links categorical features (bool type actually) to text documents. The categorical features are answers to questions. Any different combination of ...
Alessio Giberti's user avatar
1 vote
1 answer
482 views

Combining two sequences for text classification

I'm doing text classification on comments posted on articles/stories. The two human-labeled classes are appropriate and not appropriate (not the same as happy/angry or any "sentiment" ...
rodrigo-silveira's user avatar
0 votes
1 answer
139 views

Are the vectorization settings considered hyperparameters in ML?

Short definition of HP: "In machine learning, a hyperparameter is a parameter whose value is set before the learning process begins. Hyperparameter optimization or tuning is the problem of ...
Siebe Albers's user avatar
1 vote
1 answer
723 views

Using POS Tags and NERs as Features for Text Classification or Sentiment Analysis

I am trying to implement text classification and sentiment analysis from the documents. I always use POS tags as features in the following way. Mike is playing football I would convert it into ...
chaitanya's user avatar
  • 131

1
2 3 4 5
13