6
$\begingroup$

I am working on Sentiment-Analysis/Opinion-Mining of Tweets, focused on Finance related tweets.

One of the biggest issues I am facing is the unability of my algorithm to detect equivalent entities (Definition in B.Liu 2012: Page 18-19) when Financial slang is used. For example, for those familiar with it I would like the following entities to be detected as equivalent after lemmatization :

  • Government-Bonds = Govies = Sovereign-Debt

  • Cash = Monetary

  • Stocks = Equities

  • FX = Forex = Currency-exchange = Foreign-Exchange

  • Bund = German-Bonds = Bundesbank 10y

  • T-Notes = US10 = Treasury-Notes = US-Govies = American-Sovereign-Debt

  • Etc...

Here are the two sides of my question:

  1. I was thinking about using some supervised learning (Naive-Bayesian-Classification) for such task, but can't find any classified set of data for training. Do you know if such dataset exists?

  2. Do you have any alternative idea regarding how to perform such task (without dataset maybe..)?

Thanks.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

you could try word2vec, which has a nice implementation in Gensim library.

It should end up having similar vectors for the abbreviation (or slang) and the full word, and certainly can be used with sent. analysis . You could train it on a large number of tweets if there is a financial set somewhere? You would then use a linear model like Log Reg / SVM etc to classify.

this is a large tweet model, but I dont know if it would have a financial corpus within it of sufficient size, but it is pretty large! http://www.fredericgodin.com/software/ 400 million tweets 4Gb file

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ @ylnor: For a financial corpus, you should scrape a finance news site like finance.yahoo.com . In addition to this, you should scrape finance forums, to obtain even more different usages and synonyms. $\endgroup$
    – Alex R.
    Nov 22, 2016 at 19:43
  • $\begingroup$ @Luke Barker, thanks a lot. Your thought was definitely the good one. Unfortunately fitting on the full wikipedia dump wasn't enough and financial slang was not understood by the model. Anyways, after retriving a lot of data (mostly financial tweets and dumps of articles from financial newspapers), word2vec now works like a charm, and most_similar for 'T-Bills' are: 'Govies', 'Bund', 'Bonds', '10y', 'treasuries', etc... Thanks a lot for your help, that took me a while but was definitely worth it! I'll forward the paper to you as soon as it's advanced enough in the publication process, thanks! $\endgroup$
    – ylnor
    Apr 11, 2017 at 10:43
  • $\begingroup$ hey that would be great - glad to be of help and thanks for replying :) $\endgroup$
    – Luke Barker
    Apr 18, 2017 at 12:23
2
$\begingroup$

You could use Kim's Character-Aware Neural Language Models https://arxiv.org/abs/1508.06615 as an alternative to word2vec. It uses a CNN over character inputs to produce a fixed size vector for each word. It can scale to previously unseen words. This feature of scaling to previously unseen words is why I prefer it over "simple" word2vec.

FastText by Facebook is also interesting (and quicker) than Kim's model.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ It looks like you have an intimate connection with at least one of your recommendations, Maxime. Unless you explicitly disclose that, users tend to take answers like this one as a form of spamming the site. $\endgroup$
    – whuber ♦
    Dec 4, 2018 at 18:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.