Applying PCA on a set of documents gives strange results in terms of the variance explained by the PCs vs the data separation I'm having when plotting the first two principle components.
Number of documents = 200 (100 Movie Reviews & 100 Restaurant Reviews) Number of features: 5000 words
All docs have been processed and normalized (e.g., stop word removal, stemming, lemmatization, etc).
vectorizer = TfidfVectorizer() X = vectorizer.fit_transform(processed_corpus) pca = PCA() principle_components = pca.fit_transform(X) # Variance explained by the first two components print(pca.explained_variance_[:2])
Variance of PC1 = 0.04216743 Variance of PC2 = 0.01481811
As noted the explained variance of the first two PCs is very small. However, when I plot the first two principal components of each sample (document) it shows a very good separation between samples that belong to two different classes.
I'm not sure if there is anything wrong with the logic I'm using?