I have a Twitter dataset that has both the tweets themselves, but also metadata about the user as number of tweets, date of creation, and other numerical values. How can I make predictions based on the whole information instead of text OR numbers?

What I did.. but I´m not sure if it´s ok is the following: Made a "bag of words", extracted the most frequent words, deleted the "tweet" column and added one column for each "Frequent word", so if the word is present in the row (one tweet) it´d be 1, if not, 0. I guess there should be a more intelligent way.


Yes, your high-level procedure works great:

  1. Map text to a vector of features.
  2. Concatenate with the rest of your numeric features and pass the whole thing to your training algorithm.

However, there are many, many ways to go about step 1, and the best way will depend largely on the context of your analysis and the task at hand.

For example, the bag-of-words approach ignores any relationships between words. That may be totally fine if your prediction task depends on the presence or absence of certain key words, but it may be insufficient if the prediction depends on key phrases or complex interactions of words, emoji, and punctuation over the entire tweet. Furthermore, you may find that infrequent words are actually important while frequent words are just filler.

To address this, you might use TF-IDF, which balances the frequency of a word (or 2- or 3-word sequence, called $n$-grams) in a particular tweet with the relative frequency of the word across all tweets, thereby reducing the impact of common filler words. But you would still have 1 column per word or $n$-gram, which will result in a very high-dimensional feature vector unless you apply some form of dimensionality reduction in addition.

At the other end of the complexity scale, a recurrent neural network, such as an LSTM, can capture complex structure and long-term dependency in your tweets in theory, but it requires large amounts of data and careful tuning of its hyperparameters in order to avoid over-fitting.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.