Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

0
votes
0answers
7 views

Aggregation across sentences in a document

I'm using deepMoji for a text classification problem. Deepmoji returns a vector of 64 emoji for twitter sized text. I'm running deepMoji on documents that are much longer and I'm wondering if there ...
0
votes
0answers
7 views

Current SOTA in text classification

I've been recently starting with text classification, yet the amount of work in this field is somewhat overwhelming. Could you please direct me towards some of the SOTA (deep) approaches for solving ...
0
votes
0answers
8 views

Enrich true signal from loads of data

Background For simplicity let's say we have an alphabet of ABC and we are looking at words that all have the same length (n = 10)...
1
vote
0answers
20 views

HDP: Gibbs sampler implementation

I am trying to recreate the model proposed by Gao et al. (2011), based on the Hierarchical Dirichlet Process proposed by Teh and al. (2005). To estimate the model (let's call it iHDP) I need to ...
0
votes
0answers
7 views

How do I normalize or weight document-feature-matrix by length of dictionary entries

What is the best practice way to normalize or weight document-feature-matrices by the length of dictionary entries. Here is some sample code. In reality, my example works with different issues e.g. ...
7
votes
4answers
2k views

Check if a character string is not random

Background Let's say we have an alphabet of A,B, C, D, then we look through some data and find a "word" which is DDDDDDDDCDDDDDD ...
1
vote
1answer
28 views

Word2Vec and PyTorch - am I approaching this correctly?

My understanding of Word2Vec is that the library allows for generation of an array of numbers that approximates the meaning of a word relative to others in a sentence. My use of Word2Vec e.g. ...
0
votes
0answers
13 views

Clustering short messages

I have a dataset of short message conversations (from 1 to 20 words). I would like to cluster the messages that were sent to me to extract the different topics that were discussed by my interlocutors. ...
0
votes
1answer
5 views

text mining - vocabulary size very large

Question: when you have create a corpus of let’s say, 10,000 documents, and the vocabulary size made for these is let’s say, 1 million, what best practices exist to either work with this type of ...
0
votes
0answers
21 views

Word embedding as input or raw text?

I'm trying to implement a neural network for text recognition and I'm a little bit confused about text inputs. The goal of the network is to classify a comment, toy example: ...
0
votes
0answers
15 views

vectorization based on sentence meaning

Does anyone know a method to vectorize a sentence such that it's vector represents something related to meaning of the sentence. For example, consider the following instance.. Suppose we have two ...
1
vote
0answers
9 views

Associate keyword from textrank algorithm in R to verbatim [closed]

I have created a frequency table using TextRank algorithm in R. While, I get the frequencies, I want to associate back the text to the verbatim in the form of a matrix so that I know from where each ...
1
vote
1answer
17 views

Group of word representations

For word representation baseline people use bag-of-words or word embedding. Here, I want to understand all approaches that can be used for word representations. For example: -Bag-of-words (tfidf, n-...
0
votes
1answer
27 views

Representation for meaning of a text document

Does anyone know a technique to represent meaning of a given text document, so that two documents having same meaning will have same representations? Note: Typical systems like vectorization methods ...
3
votes
2answers
48 views

Intuition behind word vector representations

How is it possible for a vector space to represent words so that it is coincident to our intuition of words? What does the orthogonality concept in such a space mean precisely? I think we can present ...
0
votes
0answers
21 views

Sequential Long-Text Classification with Recurrent and Convolutional Neural Networks

I am thinking to build a model for predicting events from news. Before I start this task I wanted to ask if someone have tried to build something like in the link(https://arxiv.org/pdf/1603.03827.pdf) ...
0
votes
1answer
24 views

Content-based document Vectorization technique

Does anyone know a content-based vectorization scheme can be used in Natural Language Processing to represent documents in vector space??
2
votes
1answer
103 views

Tf-idf for text classification: On what should IDF be calculated?

The TF-IDF value of a word specifies how important a word for each document is. My setting is any text classification where one has multiple documents of with different classes: Let's take a lot of ...
0
votes
0answers
17 views

ANOVA on TF-IDF scores?

I'm analyzing three categories of texts, and looking to use TF-IDF scores to highlight differences in the importance of different words between the three categories. Would it be acceptable to conduct ...
0
votes
0answers
8 views

TFIDF Weighting With Multiple Categorized Documents

I'm doing keyword extraction on about 300.000 documents. The documents are job advertisement. Im doing; 1- Preprosess the job description 2- Use sklearn tfidfVectorizer with min 1 max 3 ngrams ...
0
votes
0answers
3 views

Understanding formulas in Guo et al. paper regarding end-to-end text classification

I finished reading End-to-End Multi-View Networks for Text Classification by Guo et al. (2017). The paper is only a few pages long and it's fairly intuitive. However, the equations are not described ...
1
vote
0answers
16 views

Unsupervised Text Clustering Project: How to get started? [closed]

I work for a manufacturing company where robust databases and data integrity have not always been a priority. I have a very messy and finite list of 13,102 tool descriptions. I need to find out how ...
1
vote
0answers
8 views

Keeping a text classifier up to date

I have built a text classifier using Naive Bayes and TF-IDF. It is a fairly weak model (~94.7 accuracy) and is more of a proof of concept before I move on to more complex methods of analysing text. ...
-1
votes
1answer
20 views

How to weight features when doing text mining?

I have a case where I'm doing text mining over a list of product titles. In particular I want to run a clustering algorithm. But I also have some information about those products that I think can add ...
0
votes
0answers
20 views

How find the most decisive sentences or words in a document via Doc2Vec?

I've trained a Doc2Vec model in order to do a simple binary classification task, but I would also love to see which words or sentences weigh more in terms of contributing to the meaning of a given ...
1
vote
0answers
12 views

Matrix of probabilities of unsupervised labeling vs annotations [closed]

Given a list of texts, annotated by topic (each text can have multiple topics), and the document-topic matrix output of an LDA (Latent Dirichlet Allocation), unsupervised model. For example: ...
0
votes
0answers
8 views

How do you find most important ngrams when the frequencies are not that diverse and reflective of information/importance?

When most "frequent" is not necessarily most "important". And when frequencies are not reflective of importance (most ngrams appear just once, twice or at best thrice) - how do you find the most "...
2
votes
0answers
47 views

R or Python Equivalent to SAS Text Miner [closed]

I am working on a project where a former member of the team used SAS Text Miner (https://support.sas.com/documentation/onlinedoc/txtminer/14.1/tmref.pdf) to complete some text mining. Unfortunately, ...
1
vote
0answers
48 views

Practically speaking, is the TF-IDF threshold universal across different corpus?

I would like to know the practical threshold of the TF-IDF (just like the practical p-value cutoff of 0.1 or 0.05 in hypothesis tests). I tried to look at it in some previous post, and some people ...
3
votes
0answers
119 views

Delta TF-IDF right choice for multi classification problem

In the paper of Martineau & Finin they describe their new approach with Delta TF-IDF . Instead of measuring how rare features are in the document, they weight these values by how biased they are ...
0
votes
0answers
14 views

can focal loss function works for text classification problem?

I am working on a relation extraction and classification problem. The data is in the form of text files. The data is imbalanced. I want to use focal loss function to address class imbalance problem in ...
0
votes
0answers
9 views

Use topic modeling to determine the topic popularity

I have a collection of documents. My end goal is to determine the popularity of the topic discussed by each document. In other words, whether (or to what extent) one document is discussing a popular ...
1
vote
1answer
22 views

How does clustering improve a language model?

This article describes a hierarchical clustering algorithm which clusters the words within a vocabulary based on their similarity, in order to improve a language model (in the article, n-grams). How ...
0
votes
1answer
20 views

natural language processing analysis

I have selected SMS Spam Collection as my dataset for natural language processing task. I have done many pre-processing tasks on dataset such as removing punctuations, spell correction, stemming, and ...
0
votes
0answers
11 views

Can we use precision and recall to evaluate text prediction model results?

I am using an LSTM model to predict/complete words from a seed input. These words or tokens represent xml markups or fields. So, as an example, I give my model a seed input : ...
1
vote
0answers
30 views

why LSTM word based model works better with random input seed than fixed input seed?

I have implemented an LSTM model that have 2 LSTM layers, a dropout layer and a dense layer for predictions. I trained my LSTM model on 1000 XML files. Each file has 4 main markups with very simple ...
0
votes
0answers
31 views

Graph-Based Semi-Supervised Learning for NLP Text Classification

I have hundreds of various job titles, where some of the titles may sound different but ultimately should be classified to be the same position (ex. Call Center, Call Center, Customer Contact Center ...
1
vote
1answer
152 views

What is the meaning of the average value of all word vectors in the sentence?

Today I saw a sentiment analysis article here. There is one piece hard to understand: Next we have to build word vectors for input text in order to average the value of all word vectors in the ...
1
vote
0answers
33 views

Anomaly detection in logs

I am working on using Machine Learning techniques to detect anomalies in the log messages. A log can be considered as "interesting" or an anomaly if it was not been seen before. (As error logs was not ...
2
votes
0answers
20 views

Classification method for biased training data

I am trying to use patent texts to make a binary classification of a large number of patents as either 1. related to automation or 2. not related to automation. I have manually classified a small non-...
1
vote
0answers
50 views

News relevancy identification algorithm - [NLP]

I have a large set of news crawled from web as the format [timestamp, news heading, news content as text] I need to filter out news that are somehow related to ...
1
vote
2answers
83 views

Strategies for incorporating feedback for a ML algorithm

I am developing a text classification problem, in which at some time points, say at the end of each week, I receive a batch of feedback from users about correctly and wrongly classified inputs. I am ...
1
vote
1answer
70 views

Evaluation of machine learning model in production

I know this question would require the knowledge of environment that the model is deployed in. So, I'm going to disclose as much knowledge as possible while being discreet. The actual question - I ...
0
votes
0answers
27 views

Comparison of classification algorithms: How can I interpret the results?

I am experimenting on a dataset of about 18,000 articles, 12000 tagged Fake and 6000 tagged Real. I'm building a fake news classifier and I'm comparing 4 classification algorithms: Multinomial Naive ...
0
votes
0answers
149 views

LDA TopicModel Alpha Parameter (Gensim)

I fit a topic model on python with 8 topics with alpha set to auto. I am trying to determine what my alpha value is for the model. the ldamodel.alpha command in gensim outputs an array, with 8 ...
0
votes
0answers
13 views

Topic assignment in a topic document matrix. How to be robust?

I have a Document-Topic matrix whose scores were produced by multiplying a Topic-Term matrix produced by non-negative matrix factorization on a Term-Term matrix and the original Document-Term matrix ...
2
votes
1answer
172 views

TF-IDF vs just TF in text classification [duplicate]

It's common to see people using a tf-idf representation of words for text classification, but I don't understand why not just use tf. Say $tf(t,d)$ is the $tf$ of term $t$ in a document $d$ and $idf(...
1
vote
0answers
210 views

Text classification using tf idf

I am developing a text classification model. At this moment I have to classify some documents in six different classes. I am using a simple approach as a starting point based on random forests over a ...
0
votes
1answer
26 views

Determine the most important documents for supervised learning

I have somewhat of a general/high level question. Assume I'm doing supervised machine learning on some text data (tweets for example) and categorizing the documents to a certain taxonomy (multi-class ...
2
votes
0answers
34 views

Word Association in text mining [closed]

I want to extract the information from text on the basis of association like. "Shahrukh Khan is the Famous actor of Bollywood, his wife name is Gauri khan. He is 52 years old. Sample output like as <...