Applying PCA - First two components explain low variance but have high data separation when plotting

Question

Applying PCA on a set of documents gives strange results in terms of the variance explained by the PCs vs the data separation I'm having when plotting the first two principle components.

Details:

Number of documents = 200 (100 Movie Reviews & 100 Restaurant Reviews) Number of features: 5000 words

All docs have been processed and normalized (e.g., stop word removal, stemming, lemmatization, etc).

Logic:

vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(processed_corpus)

pca = PCA()
principle_components = pca.fit_transform(X)

# Variance explained by the first two components
print(pca.explained_variance_[:2])

Variance of PC1 = 0.04216743 Variance of PC2 = 0.01481811

As noted the explained variance of the first two PCs is very small. However, when I plot the first two principal components of each sample (document) it shows a very good separation between samples that belong to two different classes.

I'm not sure if there is anything wrong with the logic I'm using?

Answer 1

Your question does not list typical preprocessing steps used in NLP which are intended to reduce irrelevant and uninformative variation.

Typical pre-processing includes stemming/Lemmatization (removing endings/cases/tenses to get the base dictionary version), removal of filler/stop words (common words that provide little insight) nor parsing. See wiki for an overview. Without these preprocessing steps we would not expect any strong components accounting for a high proportion of variation. You appear to only analyse the reviews for term frequency and inverse document frequency because

Tfidfvectorizer includes the following defaults so these settings apply unless explicitly set otherwise:

preprocessor=None, tokenizer=None, analyzer=’word’, stop_words=None, Tfidfvectorizer

Try implementing at least the stop_words option and see what difference that makes.

Irrespective of your preprocessing, what your results suggest is that there is one strong covariance structure within each type of review that is distinct from the other type, but after that covariance structures interrogated to review for dominate.

In other words, most words are equally likely to appear in each type of review and be used in the same way. Group separation will appear when either words are used uniquely (appear in one type of review and not the other) or in different contexts (so the word is associated with different companion words in each context).

A question that naturally arises for me is what happens to reviews of films that have a focus on food? Any restaurant reviews of movie/literary/theater themed restaurants?