I'm working on tweet classification. To classify tweets I performed text preprocessing tasks such as lemmatization, stopword removal, and punctuation removal before tweet classification. But after preprocessing several preprocessed tweets are similar(duplicates). Is it required to delete duplicate records after the tweet preprocessing in tweet classification? If I remove duplicate values of preprocessed text, the data set is reduced from 10000 to 5000. My collected tweets contain tweets like: url1 hashtag1, url2 hashtag2, url3 hashtag1, url4 hashtag2, url5 hashtag2.
So in my preprocessing steps, all URLs are normalized into URL. So some tweets are similar after preprocessing. But for my classification, the normalized word URL is an important one. If I removed the duplicate values then the f1-score of class 1 is reduced from 94% to 89%.

  • 1
    $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Oct 22, 2022 at 14:32
  • $\begingroup$ I just saw the new explanation you added to your question, but the problem is still not entirely clear to me. Are they tweets that only contain urls? If not, can you give a couple of examples of what your dataset actually looks like -in particular examples of duplicates? As I see you're new to stackexchange, here's how to add a table to your question: meta.stackexchange.com/questions/356997/… $\endgroup$
    – J-J-J
    Oct 23, 2022 at 7:22

1 Answer 1


It is really strange that your dataset is reduced exactly by 50% when you remove duplicates. You should double-check if some bug in your code generates these duplicates out of nowhere.

If you think your code is OK, then compare the original tweets that are detected as duplicates (that is, the tweets before lemmatization, removal of stopwords, etc.). The preprocessing step could have removed some features useful to your use case. For example, depending on what you are trying to classify, it might be relevant to keep punctuation or some specific stopwords.

Also, consider using additional features, like the time when the tweets were posted, the user who posted it, their number of retweets or likes, the number of followers the user has, etc.

Popular tweets are often plagiarized, so depending on how you retrieved these tweets initially, it is not necessarily wild to have a couple of real duplicates in your dataset (and you probably shouldn't remove them). But again, 50% is a really suspicious high rate of duplicates, so you should really find out why is that.

It is really difficult to give more advice, without more details about what you are trying to classify, how you retrieved these tweets, and what is the problem you're trying to solve.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.