I am analyzing a corpus of very short texts (3-7 words long). The original unigram and bigram features ($n$ dimensions) are transformed into topic features using a custom LDA (Latent Dirichlet Allocation) algorithm. In other words, the custom LDA algorithm (adapted for short texts) performs the dimensionality reduction. The number of features $m<<n$. The new features are centered and standardized before analysis.

The ultimate goal is the classification but I am attempting to understand better the data by using different tools. One of such tools is to compute the covariance matrix of the new features, compute its eigenvalues and plot them in descending order. If doing PCA, the plot would typically reveal some $k<m$ top eigenvectors with a sharp decline in eigenvalue. Under linearity assumption, there are just small number of directions that might explain the data. The plot would look like this [![enter image description here][1]][1]

However, on my data, the plot looks slightly different. There is a rapid eigenvalue drop but tending to 1, not zero. Then after a plateau, it sharply drops to 0.

[![enter image description here][2]][2]

I am having difficulties understanding this plot. It looks like there are ~20 dominant eigenvectors explaining the most important directions of variance. After this "knee", there is a long plateau from ~40-140 where the eigenvalue does not change much around 1. Are those directions capturing noise? [1]: https://i.sstatic.net/lqEbm.png [2]: https://i.sstatic.net/kFunx.png

I am analyzing a corpus of very short texts (3-7 words long). The original unigram and bigram features ($n$ dimensions) are transformed into topic features using a custom LDA (Latent Dirichlet Allocation) algorithm. In other words, the custom LDA algorithm (adapted for short texts) performs the dimensionality reduction. The number of features $m<<n$. The new features are centered and standardized before analysis.

The ultimate goal is the classification but I am attempting to understand better the data by using different tools. One of such tools is to compute the covariance matrix of the new features, compute its eigenvalues and plot them in descending order. If doing PCA, the plot would typically reveal some $k<m$ top eigenvectors with a sharp decline in eigenvalue. Under linearity assumption, there are just small number of directions that might explain the data. The plot would look like this

However, on my data, the plot looks slightly different. There is a rapid eigenvalue drop but tending to 1, not zero. Then after a plateau, it sharply drops to 0.

I am having difficulties understanding this plot. It looks like there are ~20 dominant eigenvectors explaining the most important directions of variance. After this "knee", there is a long plateau from ~40-140 where the eigenvalue does not change much around 1. Are those directions capturing noise?