I am analyzing a corpus of very short texts (3-7 words long). The original unigram and bigram features ($n$ dimensions) are transformed into topic features using a custom LDA (Latent Dirichlet Allocation) algorithm. In other words, the custom LDA algorithm (adapted for short texts) performs the dimensionality reduction. The number of features $m<<n$. The new features are centered and standardized before analysis.

The ultimate goal is the classification but I am attempting to understand better the data by using different tools. One of such tools is to compute the covariance matrix of the new features, compute its eigenvalues and plot them in descending order. If doing PCA, the plot would typically reveal some $k<m$ top eigenvectors with a sharp decline in eigenvalue. Under linearity assumption, there are just small number of directions that might explain the data. The plot would look like this enter image description here

However, on my data, the plot looks slightly different. There is a rapid eigenvalue drop but tending to 1, not zero. Then after a plateau, it sharply drops to 0.

enter image description here

I am having difficulties understanding this plot. It looks like there are ~20 dominant eigenvectors explaining the most important directions of variance. After this "knee", there is a long plateau from ~40-140 where the eigenvalue does not change much around 1. Are those directions capturing noise?

  • 2
    $\begingroup$ That has the same shape as the plot in [this](stats.stackexchange.com/questions/87032/… xponential-decay) great answer: "any high-dimensional real data are highly contaminated by noise, so the bulk of the eigenspectrum shows the spectral behaviour of pure noise." $\endgroup$
    – denis
    Mar 21 '18 at 11:28
  • $\begingroup$ Perhaps the "smooth" shape of the curve was due to the features were transformed with LDA? $\endgroup$ Oct 9 '20 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.