Although it is often calculated differently, my intuitive understanding of PCA arises from its definition as the eigendecomposition of the sample covariance matrix. I have recently become aware of various popular methods for improving estimation of the covariance matrix (e.g. Donoho et al. 2013 Optimal Shrinkage of Eigenvalues in the Spiked Covariance Model). Indeed, the Wikipedia page for Estimation of Covariance Matrix says:
Estimates of covariance matrices are required at the initial stages of principal component analysis and factor analysis, and are also involved in versions of regression analysis that treat the dependent variables in a data-set, jointly with the independent variable as the outcome of a random sample.
I frequently find myself performing PCA of data matrices where the number of samples (n) is comparable or less than the number of variables (p), which is precisely the case where the estimation of sample covariance matrix can be improved using shrinkage or other techniques. My goal in these situations is dimensionality reduction so as to find patterns in the data (e.g. in the famous Iris dataset, PCA reveals three differnt kinds of flowers). If I were to improve the estimation of the covariance matrix, would PCA then give a "better" understanding of the structure in the data (e.g. the groups would separate nicer)? Are there any examples where shrinkage is very useful in dimensionality reduction using PCA?