I have trying to optimize the run time for my app which maps in 2D the text data according to the importance and weight of the words in each document.
However, given the size of my text data of around 100K documents, the processing time is boringly long. I describe below my procedure step by step and I am looking forward to get a feedback on how to reduce the run-time.

First, I vectorize each of my documents on the words they contain using a stoplist.

Second, I run a K-Means to create and set up my different clusters and we get the weights for each word based on their weight in their cluster. So we each document, we assign an ID and then for each word in that document we assign a weight calculated using the cosine similarity.

Third, we run a TF-IDF to get the list of the important words for each cluster.

Fourth, we then use a matrix similarity on each document in order to know the distance between others using the cosine similarity.

And lastly, we reduce the high-dimensional data using T-SNE to map it on 2D plots.

Any idea how to reduce the processing time in here?

  • $\begingroup$ Cross-posted at datascience.stackexchange.com/q/13932/3361 - please decide which site you want the question on. $\endgroup$ – Scortchi Sep 12 '16 at 9:29
  • $\begingroup$ @Scortchi i deleted the other one. $\endgroup$ – cplus Sep 12 '16 at 19:17
  • $\begingroup$ Thanks. If you get no joy here & want to move it just ask. $\endgroup$ – Scortchi Sep 13 '16 at 8:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.