Questions tagged [nltk]
NLTK stands for Natural Language ToolKit, a Python-based platform for working with human language data.
21
questions
0
votes
0
answers
11
views
How to split a paragraph into sentences where there is no punctuation, including full stops, commas, etc [closed]
i want to split the paragraphs into sentences where there is no punctuation, including full stops, commas, etc.
Eg: "My name is Brayan I am 12 years old"
need to be split as
My name is ...
0
votes
1
answer
349
views
Perplexity for different n-gram models
I'm training a Lidstone Model with different n-gram sequences to see witch one is the best (2-gram, 3-gram, 4-gram, etc) in the same text database.
When I give all these models an unseen text sample ...
- 1,420
1
vote
1
answer
2k
views
Should I remove stopwords before generating n-grams?
I'm wondering if the stopwords are useful in n-gram or it should be removed before generating n-gram.
I would like to know best practices on extract features of text. I'm currently using nltk.
- 113
1
vote
2
answers
1k
views
NLTK: odd outputs from bleu_score
For machine translation purposes I use bleu score, which seems to be the validation mechanism of choice (used in the sutskever 2014 sequence-to-sequence).
The purpose is to get as high bleu as ...
- 425
0
votes
1
answer
264
views
When to use documents vs. sentences for Word2Vec?
I have a collection of words from different communities. Each community has a different way of using language and will provide a different word embedding. I can concatenate the sentences from the ...
- 3,379
0
votes
1
answer
3k
views
Any way to handle plurals in NLP without stemming words [closed]
I am playing around with topic modeling, and I notice a lot of related plural and singular words in my corpus (e.g. 'champion' and 'champions' are both found).
I gave the porter stemmer in NLTK a ...
CommunityBot
- 1
23
votes
2
answers
14k
views
How is the .similarity method in SpaCy computed?
Not Sure if this is the right stack site, but here goes.
How does the .similiarity method work?
Wow spaCy is great! Its tfidf model could be easier, but w2v with only one line of code?!
In his 10 ...
CommunityBot
- 1
2
votes
2
answers
1k
views
Tagging of tweets using NLTK
Is there a method to perform tagging of tweets using NLTK? The pos_tag() function gives incorrect results on twitter data (which uses textese):
...
- 101
1
vote
1
answer
2k
views
nltk multi_kappa (Davies and Fleiss) or alpha (Krippendorff)?
I'm using inter-rater agreement to evaluate the agreement in my rating dataset. I have a set of N examples distributed among M ...
- 136
0
votes
0
answers
881
views
NLTK BigramAssocMeasures.pmi is give same score for all bigrams
I am trying to use BigramAssocMeasures PMI to find the most import bigrams however it's giving all Bigrams the same score, so I end up with a list in alphabetical order when I use ...
CommunityBot
- 1
0
votes
0
answers
333
views
Sentiment Analysis - Emoticon
I am working on an NLP project, my objective is to compute a Sentiment Analysis over short text message.
So far, I did not come up with a solution to account the emoticons. I want to use a machine ...
- 107
0
votes
1
answer
2k
views
NLP various probabilities estimators in nltk
I saw there are many types of probabilities in nltk:
MLE, ELE, Laplace, Heldout, KnereserNey, Lidstone, Random, WittenBel..
What is the exact difference between them and when should I use each?
My ...
- 283
0
votes
0
answers
65
views
Naive Bayes Assignment of Feature Probability
I'm using the .show_most_informative_features() function from NLTK's Naive Bayes to generate features to be used with a lexicon. In the case of my binary-classification problem, these features are ...
- 111
0
votes
0
answers
185
views
Handling False Positive in a Classifier
Suppose I have the following code of an NLTK Naive Bayes Classifier.
It is a toy example of a sentiment analysis implementation.
...
- 509
0
votes
1
answer
73
views
How to classify text when having very little training data
I have a dataframe as follows:
New_Text | New_Score
review1 | Positive
review2 | Negative
review4 | Positive
... and so on.
I want to create a model ...
- 5,786
6
votes
1
answer
5k
views
Difference between Log Entropy Model and TF-IDF Model?
I would like to understand what are the differences/advantages in using TF-IDF or the Log Entropy model for represeting documents and queries in an information retrieval system using diferent weights.
...
- 1,599
1
vote
2
answers
133
views
Why does my sentiment analyser only output 1 label?
I am trying to do sentiment analysis on a corpus of product reviews. My corpus contains 50,000 samples, of which I take 70% for training and 30% for testing. I discretized the 5-star rating to 3 ...
- 20.1k
2
votes
2
answers
2k
views
Is there any package in R/Python which can analyze Pos./Neg sentiment of whole review? [closed]
I'm a newbie here in the forum and new to text analytics using Python and R. My question is somewhat similar to Is there a better approach than counting positive-negative words in sentiment analysis?
...
CommunityBot
- 1
2
votes
1
answer
579
views
How may I convert Perplexity to F Measure
In the practice of Machine Learning accuracy of some models are determined by perplexity, (like LDA), while many of them (Naive Bayes, HMM,etc..) by F Measure.
I like to evaluate all the models with ...
- 2,711
3
votes
0
answers
460
views
Cross Co-occurrence between two corpora
I've looked around for a solution to this problem specifically in nltk, quite a bit but couldn't find much help either on SO or elsewhere.
My problem is as follows:
I have a set of aligned pairs of ...
- 569
1
vote
1
answer
251
views
How to find similar document with new vocabulary
I am working on a problem of finding similar documents. I am using a Tf-Idf based vector space model representation of documents and it gives me good results.
However when I encounter a document ...
CommunityBot
- 1