Questions tagged [tf-idf]
In information retrieval, tf–idf, short for term frequency–inverse document frequency, is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus.
83
questions
1
vote
0
answers
78
views
Why does scikit-learn smooth IDF by adding 1 to both the numerator and demominator?
Scikit-learn says:
If smooth_idf=True (the default), the constant “1” is added to the numerator and denominator of the idf as if an extra document was seen containing every term in the collection ...
- 718
2
votes
1
answer
142
views
what is the scale of TF-IDF results?
I did a classification of review text data using the ANN model. As we know, text data must be converted into numeric first before entering the classification stage. To change this I used the TF-IDF ...
- 117
0
votes
0
answers
27
views
Adjusted TF-IDF where many terms appear in every document
Struggling with something so hoped the brilliant minds of the internet could help me out.
I have a large dataset of job postings from which I have extracted the skill demand (no. of times a skill is ...
- 3
0
votes
1
answer
84
views
How to interpret NearestNeighbor results obtained using cosine similarity for tf-idf vectors
Why is the top result obtained using cosine similarity extremely close to 0 not the expected 1?
That implies complete orthogonality.
Data:
100k documents/rows with 2000 features(TF_IDF values of ...
- 111
0
votes
0
answers
31
views
Combine TF-IDF with Supervised Learning for Semantic Similarity
I use TF-IDF to compute text similarity scores. It correctly identifies words, that are unique to a document in comparison to the whole corpus. In my case project names and codes are a strong ...
- 143
0
votes
0
answers
21
views
Should I do documents transformation at once or a pair at a time for auto grading with cosine similarity?
I'm developing auto grading essay that compares the similarity between the answer key and student answer with cosine similarity. This one is written in php. Let's say in a course there are 30 - 100 ...
0
votes
0
answers
85
views
Gensim NMF interpretation and output
In these days I am studying and applying Gensim NMF.
Looking at the documentation I would like to understand how it works, in their example here they have the following matrices:
W is a word-topic ...
- 111
0
votes
0
answers
30
views
Is TF-IDF scored between 0 adn 1?
From the article The TF*IDF Algorithm Explained it states.
TF-IDF is scored between 0 and 1.
Is this a mistake?
Out of 10k documents I have a word that is only found in one document.
In that one ...
- 111
1
vote
0
answers
62
views
Input for LSI, LDA and HDP?
It seems that the Latent Semantic Indexing (LSI or Latent Semantic Analysis, LSA) can take as input information frequency data (such as TfIdf-weighted space), e.g., according to this documentation ...
- 31
1
vote
1
answer
92
views
How to justify logarithmically scaled frequency for tf in tf-idf?
I am studying tf-idf (term frequency - inverse document frequency). The original logic for tf was straightforward: count of term t / number of total terms in the document.
However, I came across the ...
- 85
1
vote
1
answer
22
views
How to extract numerical features that can separate well documents belonging to two different classes?
I have a group of texts belonging to two different classes. I would like to extract numerical features that can separate well the two classes.
Right now I implemented a classic TF-IDF with a document ...
- 21
0
votes
0
answers
49
views
Unsupervised clustering with a categorical with tens of thousands of levels
I need to perform a clustering analysis of a medical claims dataset to identify anomalous healthcare providers.
My dataset contains a variable called diagnosis code ...
- 639
0
votes
1
answer
127
views
How do you use a TF-IDF matrix to score text similarity?
I'm trying to match strings of words with a website which has bulletpoints from all of the URL's I'm interested in whose text is most similar to it. The way I thought of doing it is to get all of the ...
- 121
0
votes
1
answer
75
views
NLP - train a model
I have a large set of large texts (around 60K texts, each one having between 100 and 30K words). Each text has 5 corresponding values (the famous big5 traits). The task is the following: for a new ...
0
votes
0
answers
139
views
Normalization on datasets with different distribution
I am having two datasets one is used for training a model and another one for testing it. The training dataset is large scale corpus of general context (parallel text) while the testing dataset ...
- 101
0
votes
1
answer
47
views
reducing overfitting does not improve performance
I'm training a multi-class classifier on text documents using a very classic (and somewhat old-fashioned) method on a data set consisting in relatively long text documents (average of 3000 tokens). ...
- 101
0
votes
0
answers
24
views
NLP: vectorizer/metric to upweight absence of frequent terms
I'm doing hierarchical clustering of documents in a corpus; there are words that occur in almost all the documents.
To define document similarity, I've used ...
- 61
0
votes
1
answer
3k
views
Calculating TF-IDF on a test set, having already built on training set
I have a need to engineer features from TF-IDF values for a downstream classification task. I (think) I have a reasonable grasp of TF-IDF as described in Sci-kit Learn documentation, but am unable to ...
- 263
0
votes
0
answers
28
views
Clustering mixed data based on text anlysis: Approach evaluation
As part of my project, I've been trying to analyse (and hopefully make some knowledgeable conclusions about) the movie database dataset, which consists of the following columns:
Movie ID - ID of a ...
0
votes
0
answers
48
views
Clustering mixed data based on text anlysis: Sparse Matrix problem
Good day/evening/any other time of the day!
As part of my project, I've been trying to analyse (and hopefully make some konwlegable conclusions about) the movie database dataset, which consists of the ...
1
vote
0
answers
2k
views
What is the base of the `log` in tf-idf?
I have seen the wikipedia's document. For idf in this document, log10 is used.
I have seen other examples of idf, using log base of e or ln.
What is the difference? Is there any significant ...
2
votes
1
answer
297
views
-1
votes
1
answer
45
views
Recommender Engine for documents VS Search engine indexing
I have a lot of books and I want to make recommendations to users based on the description and the title of those books.
I think that one way is to preprocess the content of the title and ...
- 119
1
vote
0
answers
546
views
PCA on TF-IDF matrix
I want to perform PCA on TF-IDF matrix, but I am not sure, should I center this matrix first or not? And should I do scaling or just centering?
- 33
1
vote
1
answer
534
views
Why does TF-IDF use the count of the most common word in the document instead of total count of all words in the document?
From what I understand, the "term frequency" ("TF" in TF-IDF) is calculated from the number of times a certain term occurs in a document vs the most common term in a document:
...
- 207
1
vote
1
answer
2k
views
How does adding/omitting lemmatization affect TF-IDF?
Lemmatization has some obvious benefits in TF-IDF, e.g. it decreases the vocabulary size.
What are some other advantages, and what are some disadvantages to lemmatizing in the context of TF-IDF?
2
votes
1
answer
2k
views
How to calculate tf-idf for a single term
I am following the tf-idf method described in this paper: Measuring, Predicting and Visualizing Short-Term Change in Word Representation and Usage in VKontakte ...
- 592
2
votes
0
answers
107
views
Alternative to tf-idf with smaller penalty on previous usage
I am using tf-idf to find words that are particularly important to individual documents. This works pretty well for my purposes. However, one area where I feel like it isn't great is how harshly words ...
- 141
2
votes
2
answers
837
views
Applying TF/IDF to non-text data?
I have a classification problem in which I am supposed to predict the end state of an object based on a set of events it experiences. There are about one thousand possible events and each object is ...
- 61
2
votes
2
answers
8k
views
Determining epsilon for DBSCAN
I'm using the method described in this paper for determining the optimal epsilon value for DBSCAN clustering in which a plot of the nearest neighbors is used:
However, the plots in the paper and ...
0
votes
1
answer
26
views
Did my text data come from two distinct distributions?
I have labeled text data from two different classes. I have calculated tfidf feature representation of all the sentences in question. I have a huge matrix where rows are sentences and columns are ...
- 289
0
votes
1
answer
527
views
High Precision and low recall score for TF-IDF when using KNN algorithm
I have twitter data which is labelled with the sentiment(Postive, Negative, Neutral) and I have evaluated the performance of Tf-Idf and Doc2Vec feature extractor using the KNN algorithm and logistic ...
1
vote
1
answer
496
views
Correct algorithm for string classification
I have a long list of DNA strings (of equal length) made of 4 letters (A,T,G,C). I want to do a binary classification on the strings. I have two basic quetsions:
I have a lot of duplicate strings ...
- 289
1
vote
0
answers
582
views
Clustering algorithms for multiple features that are arrays/lists
my issue is that I want to cluster data from a specific context.
The dataset was 3 different texts that cannot be concatenated.
Now I have the following "data-object" in Python:
It is an array with ...
- 11
0
votes
0
answers
277
views
What is the maximum number of features in Logistic Regression Problem
I was doing Text classification(binary) hosted on kaggle with approx 1.3 millions observations.
My approach is to use Logistic Regression after computing the TF-IDF matrix with n-grams = 1:3.
With ...
- 33
0
votes
1
answer
814
views
Negative values in word vectorizations
I am currently in the middle of reading Applied Text Analysis with Python by Bengfort, Bilbro, and Ojeda, and encountered a sentence that I've struggled to wrap my head around. In the section ...
- 251
2
votes
1
answer
547
views
TF-IDF String to Vector Weka bias
For example, let's say I have a text dataset like:
"words text etc",label
"words text etc",label
"words text etc",label
If I ...
- 91
1
vote
1
answer
2k
views
How to combine two tfidf sparse vectors
Say that I have two document collections that I have created a tf-idf sparse vector for each one using TfidfVectorizer. How could I combine those two vectors into one that would resemble the tfidf of ...
3
votes
1
answer
2k
views
Tf-idf for text classification: On what should IDF be calculated?
The TF-IDF value of a word specifies how important a word for each document is. My setting is any text classification where one has multiple documents of with different classes:
Let's take a lot of ...
- 81
1
vote
0
answers
687
views
Practically speaking, is the TF-IDF threshold universal across different corpus?
I would like to know the practical threshold of the TF-IDF (just like the practical p-value cutoff of 0.1 or 0.05 in hypothesis tests).
I tried to look at it in some previous post, and some people ...
- 133
3
votes
0
answers
440
views
Delta TF-IDF right choice for multi classification problem
In the paper of Martineau & Finin they describe their new approach with Delta TF-IDF . Instead of measuring how rare features are in the document, they weight these values by how biased they are ...
- 81
1
vote
0
answers
382
views
Why is the size of fitted truncated svd model is so big?
I have a dataset with tfidf matrix of shape (200000, 565000). I am fitting truncated svd of 500 dimensions from sklearn onto it and pickling the resulting svd object for later use. The pickle file is ...
- 11
1
vote
1
answer
4k
views
How to improve performance for LDA?
I am running LDA on health-related data. Specifically I have ~500 documents that contain interviews that last around 5-7 pages. Other than that, I cannot really go into the details of the data due to ...
- 990
2
votes
1
answer
4k
views
TF-IDF vs just TF in text classification [duplicate]
It's common to see people using a tf-idf representation of words for text classification, but I don't understand why not just use tf.
Say $tf(t,d)$ is the $tf$ of term $t$ in a document $d$ and $idf(...
- 521
1
vote
0
answers
471
views
Text classification using tf idf
I am developing a text classification model. At this moment I have to classify some documents in six different classes. I am using a simple approach as a starting point based on random forests over a ...
- 223
0
votes
1
answer
420
views
Using k fold cross validation gives lower results than without using it
I have implemented text classification in the sentence level by following through this tutorial. I have used tf-idf and NB & SVM as shown in the tutorial. The code is working fine with my dataset.
...
- 101
1
vote
0
answers
127
views
How to compute tf-idf for sequential k means?
I am trying to run the sequential K means algorithm as described here on a corpus using tf-idf as a vectorized representation of my documents. I do this because I don't have access to all of my ...
- 111
0
votes
2
answers
653
views
Use tf-idf for QnA
I have a corpora of queries and answers where the queries are the title of different documents and the answers are a short description of them. E.g:
Title: African Civilian Deaths
Description:...
- 101
0
votes
0
answers
1k
views
Accuracy increases on decreasing the percentage of training data with stable precision, recall and F-score
I am currently working on a classification problem using tf-idf and Naive Bayes for two classes A and B. I have randomly shuffle the dataset before implementation, and I was experimenting with the ...
- 101
1
vote
0
answers
92
views
TF-IDF on a sub-corpus
I'm analyzing newspaper articles, some are 2 or 3 pages long, some a few lines long.
My Corpus is made of a few thousands of dated articles spanning over a few ...
- 123