0
$\begingroup$

I'm trying to do sarcasm detection on Twitter data to replicate the results mentioned in this paper. Binary classification problem. For that I used a separate set of unlabeled tweets to create the embedding matrix using Word2Vec model. Before doing that I preprocessed the unlabeled data and removed the rare words as mentioned in the paper. Code is as follows:

model = Word2Vec(df_hing_eng['tweet_text'], vector_size=300, window=10, hs=0, negative = 1)
embedding_size = model.wv.vectors.shape[1]

Next I fit a tokenizer on this unlabeled data:

tok = Tokenizer()
tok.fit_on_texts(df_hing_eng['tweet_text'])
vocab_size = len(tok.word_index) + 1

Next, I created the embedding matrix as follows:

word_vec_dict={}
for word in vocab:
    word_vec_dict[word]=model.wv.get_vector(word)

embed_matrix=np.zeros(shape=(vocab_size,embedding_size))
for word,i in tok.word_index.items():
    embed_vector=word_vec_dict.get(word)
    if embed_vector is not None:  
        embed_matrix[i]=embed_vector 

Now, I'm using a separate set of labeled tweets to be used as training and test data (for the DL models). I used the same preprocessing steps as the unlabeled data and removed the same rare words we found in the unlabeled data. Now I find the maximum length of all tweets in the labeled data.

maxi = -1
for row in df_labeled.loc[:,'tweet_text']:
    if len(row)>maxi:
        maxi = len(row)

After that I used the tokenizer, that I fit on the unlabeled data, to create the word indices for the labeled data as follows:

encoded_tweets = tok.texts_to_sequences(df_labeled['tweet_text'])

Now I padded the labeled data to the length of the maximum tweets among the labeled data.

padded_tweets = pad_sequences(encoded_tweets, maxlen=maxi, padding='post')

Finally, I split the labeled data into training and test data as follows,

x_train,x_test,y_train,y_test=train_test_split(padded_tweets, df_labeled['is_sarcastic'], test_size=0.10, random_state=42)

Is there any data leakage anywhere from training to test data or any other problem? Almost all of my DL models are giving more than 90% accuracy contrary to the original paper which reported a maximum of 75% accuracy. The codes for DL models were written by the authors of the papers. I used the same parameters as they mentioned.

The tokenizer was actually fit on a completely different unlabeled data that is absolutely separate from (labeled) training and test data.

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.