# Dimension reduction - word embeddings as inputs for a time series model (LSTM) [closed]

As a part of my Master Thesis I plan to use news headlines as an input feature for a time series model that predicts the daily trend of Bitcoin returns (1= positive return, 0= negative return). Besides the headlines, I also ulilize other Macro economic variables possibly related to Bitcoin, which come as integers in a daily frequency (e.g. interest rates, etc.).

So far I scraped around 16,000 news headlines over the past 5 years (8-10 headlines per day). Each headline consists of max 30 words. The headlines are cleaned (lower case, punctuation and number removal, stop word removal and lemmatized) and tokenized on a word level.

The first step in order to use the headlines as an input is to represent the raw text as vectors. After quite some research I chose word embedding algorithms to do the job. I am working in R and tried a couple approaches of implementing word2vec or GloVe in order to train the word vectors. Finally, I trained the word vectors using the GloVe algorithm implemented in the “text2vec” R package.

Assuming, I now have 50 dimensional word vectors for all words in the news headlines.

1. How do I retrieve the original structure of the news headlines and bring the word vectors in a form that is compatible with the other input features (Macro variables)?

My thought were: The resulting data frame should look like a classic time series or xts object (date in rows, features in columns). Since I have max 10 headlines per date, every headline has max 30 words and every word is a 50-dimensional vector, the news headlines for one date have to be represented through 10*30*50=15,000 columns.

1. Is that reasoning correct?

In case it is: After deleting all 'only zero' columns the dimensionality shrunk down to 8,700 columns which is still way too high. A PCA (90% of the explained variance is contained) reduced the dimensionality to around 1,000 columns. However, my initial goal was to have even a lower dimensional representation of the news headlines per date.

1. How do I achieve this without losing too much information? Might doc2vec be an option instead of word embeddings like word2vec or GloVe?

Your help and advice is much appreciated. In case you have further questions or need more details/context, just let me know.

*Edit: I deleted redundant parts of the question and narrowed it down to be more specific. I hope it is specific enough so that the question can be opened again.

## closed as too broad by Michael R. Chernick, kjetil b halvorsen, S. Kolassa - Reinstate Monica, Peter Flom - Reinstate Monica♦Mar 16 '18 at 14:57

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

• I edited the question and hope it is now specific enough to get re-opened :) – hokage555 Mar 23 '18 at 17:03
• Part of this looks like a request for help w/ R code (which is off topic here), & the rest still seems too broad to be well answered in a couple paragraphs. – gung - Reinstate Monica Mar 24 '18 at 18:08

I think it would keep your code simplest if you use the Keras Embedding to handle things for you:

 model = Sequential()



It's not documented very well, but Embedding seems to learn a word2vec-style embedding, based on an input of token numbers. In the example, above, for each token it takes as input a 1-wide token number vector, and will output a 256-wide embedded vector. Note also the mask_zero, which says that Embedding -- and (many) other layers -- should ignore token number 0. So left-pad your input token vector with 0's. (Of course, you don't want to try to use 0 as an actual token number.)

The downside is that the Embedding expects a maximum token number (max_features) and it's only going to learn an embedding based on your training set. So you can't really do something like Transfer Learning with a pre-done, large-scale word2vec embedding.

• Thanks for the answer. I would be interested in using Keras and the Embedding function to train word vectors. So far I was using the GloVe algorithm implemented in the text2vec R package. However, getting the word vectors was not my main question. I re-formatted my initial post so that the individual questions pop out better. Do you also have any ideas on these? – hokage555 Mar 15 '18 at 20:19