0
$\begingroup$

I am trying to build an LSTM neural network to do sentiment analysis on twitter feeds. The dataset I use contains ~1.5M twitter feeds with either positive or negative sentiment (the tweets were ranked by the emojis they use). Additionally the tweets are made uppercase to simplify learning.

The reason I use LSTM is that I hope it will learn nuances between different words which are possibly misspelled etc., also it simplifies pre-/postprocessing of data when using the classifier.

I've been trying a few different sets of hyperparams but it seems to max out at 60% accuracy, which is not really what I hoped for. In the last iteration I added the third LSTM layer to the network but now training is so slow on my GTX970 GPU so I'm considering aborting since there is not even a hint of improvement even after an hour.

In which ways could I either improve the data, or redesign my network so that it still can understand weird spellings of words but still train at a decent rate? Other more traditional classifiers seems to be able to get 70% to 80% accuracy for the same dataset.

Here is the code

# Training parameters
checkpoint_path      = './checkpoint'
best_checkpoint_path = './best_checkpoint'
max_checkpoints      = 10
tensorboard_dir      = './tensorboard'

# Import `data` from file
filename = "training.1600000.processed.noemoticon.csv"
columns  = ['polarity','id','date','query','user','text']

import os, pandas as pd, numpy as np
"""
0 - the polarity of the tweet (0 = negative, 2 = neutral, 4 = positive)
1 - the id of the tweet (2087)
2 - the date of the tweet (Sat May 16 23:58:44 UTC 2009)
3 - the query (lyx). If there is no query, then this value is NO_QUERY.
4 - the user that tweeted (robotickilldozr)
5 - the text of the tweet (Lyx is cool)
"""

df = pd.read_csv(filename, names=columns)
df = df[['polarity','text']]    # Filter out relevant columns
df = df[df.polarity != 2]       # Do not consider neutral polarity
df = df.sample(frac=1)          # Randomize order
data = np.array(df.as_matrix()) # A matrix is enough

# Preprocess the `data`
from tflearn.data_utils import to_categorical, pad_sequences

test_ratio = 0.1
n_train = int(len(df)*(1 - test_ratio))
n_test  = int(len(df) - n_train)

X_train = map(lambda string: map(lambda string2: ord(string2.upper()), string), data[:n_train,1])
X_train = np.array(pad_sequences(X_train, maxlen=140, value=0), dtype=int)
X_train = np.reshape(X_train, (n_train, 140, 1))

Y_train = map(lambda x: 0 if x == 0 else 1, data[:n_train,0])
Y_train = to_categorical(Y_train, nb_classes=2)

X_test = map(lambda string: map(lambda string2: ord(string2.upper()), string), data[n_train:,1])
X_test = np.array(pad_sequences(X_test, maxlen=140, value=0), dtype=int)
X_test = np.reshape(X_test, (n_test, 140, 1))

Y_test = map(lambda x: 0 if x == 0 else 1, data[n_train:,0])
Y_test = to_categorical(Y_test, nb_classes=2)

# Build model
import tflearn

net = tflearn.input_data([None, 140, 1])
net = tflearn.lstm(net, 128, return_seq=True, dropout=0.8)   # Borde denna vara 128?
net = tflearn.lstm(net, 64, return_seq=True, dropout=0.8)    # <= ny
net = tflearn.lstm(net, 32, dropout=0.8)                     # <= viktig
net = tflearn.fully_connected(net, 16, activation='softmax') # <= viktig
net = tflearn.fully_connected(net, 2, activation='softmax')
net = tflearn.regression(net, optimizer='adam', learning_rate=0.01, loss='categorical_crossentropy')

# Fit model
model = tflearn.DNN(net, tensorboard_verbose=3, checkpoint_path=checkpoint_path,
                    best_checkpoint_path=best_checkpoint_path, max_checkpoints=max_checkpoints,
                    tensorboard_dir=tensorboard_dir)

model.fit(X_train, Y_train, validation_set=(X_test, Y_test), show_metric=True, n_epoch=100, batch_size=1024)

And the training set is available here

$\endgroup$
  • $\begingroup$ You can try to use a dictionary of correct words and then correct incorrect words to correct words based on distance between words (number of different letters) $\endgroup$ – keiv.fly May 30 '17 at 21:28
  • $\begingroup$ @keiv.fly that has nothing to do with this question. He is doing sentiment analysis, not mistake detection. $\endgroup$ – Thomas W May 31 '17 at 15:55
  • $\begingroup$ @aerkenemesis is there a reason you are filtering neutral polarity ? $\endgroup$ – Thomas W May 31 '17 at 15:58
  • $\begingroup$ @ThomasW I was talking about data transformation before final analysis. In my experience it can significantly improve fitness of the model. And Google does it in their search engine which allows me to type search requests faster. The aim of Google is not mistake detection it is just an advanced technique in text analysis. $\endgroup$ – keiv.fly May 31 '17 at 17:31
  • $\begingroup$ @ThomasW it is unclear what neutral polarity means (lack of emojis?), also there are no tweets with neutral polarity in the dataset so I also want to make it clear that it is not used for training. $\endgroup$ – aerkenemesis May 31 '17 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.