I'm trying to do sarcasm detection on Twitter data to replicate the results mentioned in this paper. Binary classification problem. For that I used a separate set of unlabeled tweets to create the embedding matrix using Word2Vec model. Before doing that I preprocessed the unlabeled data and removed the rare words as mentioned in the paper. Code is as follows:

model = Word2Vec(df_hing_eng['tweet_text'], vector_size=300, window=10, hs=0, negative = 1)
embedding_size = model.wv.vectors.shape[1]

Next I fit a tokenizer on this unlabeled data:

tok = Tokenizer()
vocab_size = len(tok.word_index) + 1

Next, I created the embedding matrix as follows:

for word in vocab:

for word,i in tok.word_index.items():
    if embed_vector is not None:  

Now, I'm using a separate set of labeled tweets to be used as training and test data (for the DL models). I used the same preprocessing steps as the unlabeled data and removed the same rare words we found in the unlabeled data. Now I find the maximum length of all tweets in the labeled data.

maxi = -1
for row in df_labeled.loc[:,'tweet_text']:
    if len(row)>maxi:
        maxi = len(row)

After that I used the tokenizer, that I fit on the unlabeled data, to create the word indices for the labeled data as follows:

encoded_tweets = tok.texts_to_sequences(df_labeled['tweet_text'])

Now I padded the labeled data to the length of the maximum tweets among the labeled data.

padded_tweets = pad_sequences(encoded_tweets, maxlen=maxi, padding='post')

Finally, I split the labeled data into training and test data as follows,

x_train,x_test,y_train,y_test=train_test_split(padded_tweets, df_labeled['is_sarcastic'], test_size=0.10, random_state=42)

Is there any data leakage anywhere from training to test data or any other problem? Almost all of my DL models are giving more than 90% accuracy contrary to the original paper which reported a maximum of 75% accuracy. The codes for DL models were written by the authors of the papers. I used the same parameters as they mentioned.

The tokenizer was actually fit on a completely different unlabeled data that is absolutely separate from (labeled) training and test data.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.