3
$\begingroup$

I have a corpus with 6040592 words and 309074 types (different words). Knowing this information it is possible to know the optimal size of bag of words vectors in order to represent phrases?

I am using a data structure like this:

{'contains(The)': True, 'contains(waste)': False, 'contains(lot)': False, ...}

To represent this:

 The movie is lovely.

Do Zipf's law could help to know how many words include in the model?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ There are approximately 200k words in the English dictionary, you got 6M..... $\endgroup$
    – Uri Goren
    Commented Feb 26, 2016 at 20:11
  • 1
    $\begingroup$ Have you removed punctuation and stop words from your corpus? Have you done any type of filtering? Where is the corpus from? Like @UriGoren said, it seems that you have too many different words. $\endgroup$
    – Armen Aghajanyan
    Commented Feb 26, 2016 at 21:29
  • $\begingroup$ Where are we on this question ? did you receive an answer ? $\endgroup$
    – Uri Goren
    Commented Mar 1, 2016 at 20:18
  • $\begingroup$ Some extra information: - The corpus is not in English - No filters were applied - My counts were approximated using the following basic bash commands: for tokens \$wc corpus, and for types \$cat corpus | tr " " "\n" | sort | uniq -c | wc -l $\endgroup$
    – alemol
    Commented Mar 1, 2016 at 21:27
  • $\begingroup$ But the question is if an optimal number of features could be assessed, It is assumed that use the entire dictionary is a trivial solution. $\endgroup$
    – alemol
    Commented Mar 1, 2016 at 21:37

2 Answers 2

Reset to default
2
$\begingroup$

Usually when using bag of words approaches, the entire dictionary is used.

The "Vector", which is more precisely called the Bag of words (BOW) if the counts of each word.

Note that stop-words, punctuation, and stemming are usually applied prior to the BOW counting, so I would expect to see much less words in the dictionary.

The Zipf distribution basically shows us the likelihood of encountering an unseen word, given our current counts.

Unseen words can be treated with Dirichlet prior, Lidstone smoothing, or even (god forbid) just filtering them out of the dictionary.

Improve this answer
$\endgroup$
1
$\begingroup$

The answer is problem-specific. Imagine that you are building a classifier of something rather generic, for example, you use the bag-of-words as a feature for a classifier that marks the sentence as talking about food vs something else. I guess that in such a case, you could discard all the stopwords and probably the vast majority of all the other words except the most common ones, and even if you were left with sentences such as "<unk> <unk> hot-dog <unk> great" it would be enough for something like naive Bayes to decide that it talks about food. Now imagine that you're building a different classifier, that detects something more esoteric and context-dependent, where many rare words can make a difference in the meaning of a sentence (sarcasm, for an extreme example). In such a case, you would need to preserve many more words for it to work.

The best way to decide if your bag-of-words is big enough is to do this empirically, by verifying how chaining its size impacts the model performance (so the metrics, but also edge cases, doesn't it make it more biased, or less fair, etc). You can start with the biggest you can buy and verify how much smaller can it be made without an unacceptable drop in performance. On the technical side, you can use things like hashing trick to shrink the large bag-of-words into smaller cardinality.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.