90

In fact, the output vectors are not computed from the input using any mathematical operation. Instead, each input integer is used as the index to access a table that contains all posible vectors. That is the reason why you need to specify the size of the vocabulary as the first argument (so the table can be initialized). The most common application of this ...


43

The general term Naive Bayes refers the the strong independence assumptions in the model, rather than the particular distribution of each feature. A Naive Bayes model assumes that each of the features it uses are conditionally independent of one another given some class. More formally, if I want to calculate the probability of observing features $f_1$ ...


32

Recently, a huge body of literature discussing how to extract information from written text has grown. Hence I will just describe four milestones/popular models and their advantages/disadvantages and thus highlight (some of) the main differences (or at least what I think are the main/most important differences). You mention the "easiest" approach, which ...


29

The answer is very straight-forward: TF-IDF can achieve better results than simple term frequencies when combined with some supervised methods. The canonical example is using cosine similarity as a measurement of similarity between documents. Taking the cosine of the angle between the TF-IDF vector representation of documents can successfully retrieve ...


26

Implementation: The topicmodels package provides an interface to the GSL C and C++ code for topic models by Blei et al. and Phan et al. For the earlier it uses Variational EM, for the latter Gibbs Sampling. See http://www.jstatsoft.org/v40/i13/paper . The package works well with the utilities from the tm package. The lda package uses a collapsed Gibbs ...


24

No. Cosine similarity can be computed amongst arbitrary vectors. It is a similarity measure (which can be converted to a distance measure, and then be used in any distance based classifier, such as nearest neighbor classification.) $$\cos \varphi = \frac{a\cdot b}{\|a\| \, \|b\|} $$ Where $a$ and $b$ are whatever vectors you want to compare. If you want ...


21

Because they are different: One does not include the other. Yes modern NLP (Natural Language Processing) does make use of a lot of ML (Machine Learning), but that is just one group of techniques in the arsenal. For example, graph theory and search algorithms are also used a lot. As is simple text processing (Regular Expressions). Note I also said "modern ...


21

As far as I know you just need to supply a number of topics and the corpus. No need to specify a candidate topic set, though one can be used, as you can see in the example starting at the bottom of page 15 of Grun and Hornik (2011). Updated 28 Jan 14. I now do things a bit differently to the method below. See here for my current approach: https://...


20

I think the most detailed answers can be found in Mehryar Mohri's extensive work on the topic. Here's a link to one of his lecture slides on the topic: http://www.cims.nyu.edu/~mohri/amls/lecture_3.pdf The problem of language detection is that human language (words) have structure. For example, in English, it's very common for the letter 'u' to follow the ...


19

the chance of finding this random seems low to me whereas finding BABDCABCDACDBACD seems less random. Why would that be? If the overall proportion of letters A...D is equal to 0.25 for each letter, and each letter is independent of the other one, then both words are exactly equally probable. If the distribution of letters differ, then of course the ...


17

This is indeed something often glossed over. Some people are doing something a bit cheeky: holding out a proportion of the words in each document, and giving using predictive probabilities of these held-out words given the document-topic mixtures as well as the topic-word mixtures. This is obviously not ideal as it doesn't evaluate performance on any held-...


17

You always need this 'fail-safe' probability. To see why consider the worst case where none of the words in the training sample appear in the test sentence. In this case, under your model we would conclude that the sentence is impossible but it clearly exists creating a contradiction. Another extreme example is the test sentence "Alex met Steve." where "...


17

You could try Shannon information: $$ H = -\sum_{i = 0}^n {P_{i}\log_{2}(P_{i})} $$ where, $P_{i} = \frac{c_{i}}{n}$, $c_{i}$ is the count of some letter $c$ in the word and $n = |{\rm word}|$. For the first word you have $H = 0.35$. In the second word you have $H = 2$. If the entropy is high, you could think of it as more random vs. another word with ...


16

Text classification problems tend to be quite high dimensional (many features), and high dimensional problems are likely to be linearly separable (as you can separate any d+1 points in a d-dimensional space with a linear classifier, regardless of how the points are labelled). So linear classifiers, whether ridge regression or SVM with a linear kernel, are ...


15

You have indeed correctly described the way to work with crossvalidation. In fact, you are 'lucky' to have a reasonable validation set at the end, because often, crossvalidation is used to optimize a model, but no "real" validation is done. As @Simon Stelling said in his comment, crossvalidation will lead to lower estimated errors (which makes sense because ...


15

Let's say you've trained your Naive Bayes Classifier on 2 classes, "Ham" and "Spam" (i.e. it classifies emails). For the sake of simplicity, we'll assume prior probabilities to be 50/50. Now let's say you have an email $(w_1, w_2,...,w_n)$ which your classifier rates very highly as "Ham", say $$P(Ham|w_1,w_2,...w_n) = .90$$ and $$P(Spam|w_1,w_2,..w_n) = ....


15

Yes, although your confusion here is understandable, since the term "sparsity" is hard to define clearly in this context. In the sense of the sparse argument to removeSparseTerms(), sparsity refers to the threshold of relative document frequency for a term, above which the term will be removed. Relative document frequency here means a proportion. As the ...


14

Bag-of-words and vector space model refer to different aspects of characterizing a body of text such as a document. They are described well in the textbook "Speech and Language Processing" by Jurafsky and Martin, 2009, in section 23.1 on information retrieval. A more terse reference is "Introduction to Information Retrieval" by Manning, Raghavan, and Schütze,...


13

Let's start by taking a look at where your expression for PMI comes from. According to this article, for a pair of outcomes $x$ and $y$, $$PMI(x,y) = \log\left[\frac{p(x,y)}{p(x)p(y)}\right]$$ This says that, in order to calculate PMI properly, you need to somehow define a rule for associating the observation of your $n$-grams with a probability. In the ...


13

Can LDA be used to detect the topic of A SINGLE document? Yes, in its particular representation of 'topic,' and given a training corpus of (usually related) documents. LDA represents topics as distributions over words, and documents as distributions over topics. That is, one very purpose of LDA is to arrive at probabilistic representation of each document ...


13

Footnote at http://arxiv.org/abs/1412.5335 (one of the authors is Tomas Mikolov) says In our experiments, to match the results from (Le & Mikolov, 2014), we followed the suggestion by Quoc Le to use hierarchical softmax instead of negative sampling. However, this produces the 92.6% accuracy result only when the training and test data are not shuffled. ...


12

Your measure of "counter productive" could be arbitrary - eg. with lots of fast memory it could be processed faster (more reasonably). After saying that, exponential growth comes into it and from my own observations it seems to be around the 3-4 mark. (I haven't seen any specific studies). Trigrams do have an advantage over bigrams but it is small. I've ...


12

+1 for topicmodels. @Momo's answer is very comprehensive. I'd just add that topicmodels takes input as document term matrices, which are easily made with the tm package or using python. The lda package uses a more esoteric form of input (based on Blei's LDA-C) and I've had no luck using the built-in functions to convert dtm into the lda package format (the ...


12

First, Just a small correction: if we have a sentence $s$ that contains $n$ words, its perplexity $\newcommand{\Perplexity}{\rm Perplexity} \Perplexity(s)$ is: $$ \Perplexity(s) = \sqrt[n]{\frac{1}{p(w_1^n)}} $$ If we want to know the perplexity of the whole corpus $C$ that contains $m$ sentences and $N$ words, we have to find out how well the model can ...


12

The problem of named entity resolution is referred to as multiple terms, including deduplication and record linkage. I doubt that it is possible to determine precisely, what software belong to some of the most popular for solving that problem. There are various approaches and algorithms can be used for named entity resolution. Therefore, software which ...


12

Latent Dirichlet Allocation (LDA) is great, but if you want something better that uses neural networks I would strongly suggest doc2vec (https://radimrehurek.com/gensim/models/doc2vec.html). What it does? It works similarly to Google's word2vec but instead of a single word feature vector you get a feature vector for a paragraph. The method is based on a ...


11

I think you have not yet understood the difference between clustering and classification. Document classification (or supervised learning) requires a set of documents and a class information for each document (example: the topic of the document). The goal of classification is to build a model which predicts the class for documents where the class (in this ...


11

I personally find the string parsing methods in Python much more intuitive than R, and chaining makes the code very readable. R: # Lowercase, remove !, tokenize string<-"This is a string!!!" newstring<-strsplit(tolower(gsub("!","",string))," ") Python: # Lowercase, remove !, tokenize string="This is a string!!!" newstring=string.lower().replace("!...


11

I don't know how it's possible to do keyword extraction with supervised learning, but I do know how to do it with unsupervised learning. There are several methods of doing this, so here they are: Hierarchical You can apply any hierarchical clustering method on the term similarity matrix directly (with any similarity function, not just cosine) In scikit-...


11

"Corpus" is a collection of text documents. VCorpus in tm refers to "Volatile" corpus which means that the corpus is stored in memory and would be destroyed when the R object containing it is destroyed. Contrast this with PCorpus or Permanent Corpus which are stored outside the memory say in a db. In order to create a VCorpus using tm, we need to pass ...


Only top voted, non community-wiki answers of a minimum length are eligible