I have trying to optimize the run time for my app which maps in 2D the text data according to the importance and weight of the words in each document.
However, given the size of my text data of around 100K documents, the processing time is boringly long. I describe below my procedure step by step and I am looking forward to get a feedback on how to reduce the run-time.
First, I vectorize each of my documents on the words they contain using a stoplist.
Second, I run a K-Means to create and set up my different clusters and we get the weights for each word based on their weight in their cluster. So we each document, we assign an ID and then for each word in that document we assign a weight calculated using the cosine similarity.
Third, we run a TF-IDF to get the list of the important words for each cluster.
Fourth, we then use a matrix similarity on each document in order to know the distance between others using the cosine similarity.
And lastly, we reduce the high-dimensional data using T-SNE to map it on 2D plots.
Any idea how to reduce the processing time in here?