I'm building a sentence classifier with a Convolutional Neural Network (CNN) architecture. I would like to do the word embedding outside of my CNN using a pre-trained model such as GoogleNews (which is based on word2vec). I'm wondering if it is worthwhile to add part-of-speech information to this model and if so, how?

I see the following options:

  1. Use just word2vec to embed words into 300-features vectors
  2. Use two channels in my CNN - one for word2vec and one for part-of-speech tag. Do I have to then embed the part of speech tag into 300 features too?
  3. Embed the part-of-speech tag to some other number of features (say 20 features) and concatenate this 20-feature vector to the word2vec vector (resulting with 320-feature vectors)

If #2 or #3 are preferable, which methods are available to embed a POS tag to a vector representation?

  • $\begingroup$ Can you give an example of part-of-speech information? I would also recommend calling a sentence a document, since you are treating each sentence as a separate document. $\endgroup$ – JoleT Oct 2 '16 at 15:24
  • $\begingroup$ I wouldn't recommend use of acronyms without defining them first, as it's not best practice. What is CNN? You might withdraw this question and go to a natural language processing or text mining forum. $\endgroup$ – JoleT Oct 2 '16 at 17:47
  • 1
    $\begingroup$ I am using parsey mcparseface (AKA syntaxnet) to extract part-of-speech information. The part of speech tags have two levels - general POS and a more specific POS. For instance "walking" - general POS is 'verb', specific POS is 'gerund verb' $\endgroup$ – zvisofer Oct 2 '16 at 17:48
  • $\begingroup$ I accidentally hit enter while typing... $\endgroup$ – zvisofer Oct 2 '16 at 17:49
  • $\begingroup$ CNN - convolutional neural network $\endgroup$ – zvisofer Oct 2 '16 at 17:53

1. Concatenating word2vec and POS features

Adding POS information to your classifier is fine. You will of course want to create a train/dev/test split, eg 5-way cross-validation, to test to what extent adding this information improves your results (it's data dependent, really depends on your data, only you can test this, using your own data).

To combine the POS and word2vec features, you can simply concatenate them. I assume when you say 'CNN', you mean '1-dimensional CNN', is that right? So your input data if you were just using word2vec features, would be something like:

[batch size][sequence length][word2vec dimensions (ie 300)]

ie, a batch size * sequence_length * word2vec_dim sized tensor. So, concatenating with the POS features, your input data tensor will become:

[batch size][sequence length][
    word2vec dimensions (ie 300) + POS dimensions (ie 20)]

ie a batch size * sequence_length * 320 sized tensor.

2. sense2vec

You might also want to check out sense2vec, from Trask et al, 2016, https://arxiv.org/pdf/1511.06388.pdf , which makes use of POS information to disambiguate word2vec embeddings:

"This paper presents a novel approach which addresses these concerns by modeling multiple embeddings for each word based on supervised disambiguation, which provides a fast and accurate way for a consuming NLP model to select a sense-disambiguated embedding. We demonstrate that these embeddings can disambiguate both contrastive senses such as nominal and verbal senses as well as nuanced senses such as sarcasm. We further evaluate Part-of-Speech disambiguated embeddings on neural dependency parsing, yielding a greater than 8% average error reduction in unlabeled attachment scores across 6 languages."

  • $\begingroup$ Is there any experience if it is advisable to also learn an embedding for the POS tags or should one just use one-hot encoding if it are not too many tags? $\endgroup$ – McLawrence Sep 20 '18 at 13:27

I understand that this is an old question, but since there's no answer yet I'll add one.

The intuition here is a well-motivated one: feed the grammatical structure to your model by using a good existing parser, so that your model doesn't have to learn to understand the grammatical structure from scratch.

Something along these lines was (somewhat recently) implemented by the SEST/SEDT model from CMU. Instead of tagging the parts of speech they actually feed in a full parse tree. The results appear to be favorable.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.