1
$\begingroup$

How do I create word vectors from a corpus where sentences are very short. e.g if the corpus contains messages from users - 'good morning', 'hello!', 'No, I can't.', 'Where?' etc. One way to resolve this might be to use pre-trained vectors, is there any other way?

Also, the main problem is what if the short sentences are a mixture of two languages, English and Hindi (Hinglish - 'nahi aa sakta'). So the corpus would have sentences like - 'Yes', 'Hi! Meeting kab hai?' 'Location kya hai?' 'When will you reach?' (As I won't get pre-trained vectors for the Hinglish words, how do I deal with them in this situation?)

How do I go about getting vectors for such sentences for any NLP task?

$\endgroup$
0
$\begingroup$

Pre-trained word vectors seems the most logical solution for your problem. Other than that, it might be worthwhile translating your non-english words to retain similarity. Services like Google Cloud offer automatic translations (like using google translate directly) accessible through typical NLP language like R or Python..

I don't believe there are pre-trained word embeddings that have been trained on different languages and made consistent, I also think this would be a very non-trivial task because of the fundamental differences between languages!

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the answer! Yeah I think pre-trained vectors are the only possible vectorisation options. Currently the task I am focusing on is Intent classification during a limited conversation. So I think I'll try out hand crafting features too as short text sentences can only be of so many types as they should be limited in the amount of information they can have due to the word count. Also I'll try to implement a method so that the counter questioning bot can funnel the user towards specific words upon which I craft the features. $\endgroup$ – Vishal Lamba Jul 27 '19 at 22:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.