Regularization methods for factor analysis (in the $nIs there any covariance matrix regularization suitable for factor analysis?
I have a data matrix where number of observations is smaller than the number of dimensions: $n<p$.
I am thinking of something like this paper which has been proposed for linear discriminant analysis. 
 A: I see that the question is old but maybe some one finds the answer still helpful.
Recently, there were quite some papers published on this, here are some references, you might want to check out:
Jung, S., & Takane, Y. (2007). Regularized common factor analysis. New Trends in Psychometrics, 1(1), 1–10.
Jung, S., & Lee, S. (2011). Exploratory factor analysis for small samples. Behavior Research Methods, 43(3), 701–9. 
Jacobucci, R., Grimm, K. J., & McArdle, J. J. (2016). Regularized Structural Equation Modeling. Structural Equation Modeling, 23(4), 55–566. 
Bai, J., & Liao, Y. (2016). Efficient estimation of approximate factor models via penalized maximum likelihood. Journal of Econometrics, 191(1), 1–18. 
Hirose, K., & Yamamoto, M. (2015). Sparse estimation via nonconcave penalized likelihood in factor analysis model. Statistics and Computing, 25(5), 863–875. 
Hirose, K., & Yamamoto, M. (2014). Estimation of an oblique structure via penalized likelihood factor analysis. Computational Statistics and Data Analysis, 79(Kaiser), 120–132.
Hirose, K., & Yamamoto, M. (2015). Sparse estimation via nonconcave penalized likelihood in factor analysis model. Statistics and Computing, 25(5), 863–875. 
Trendafilov, N. T., & Adachi, K. (2015). Sparse Versus Simple Structure Loadings. Psychometrika, 80(3), 776–790.
Trendafilov, N. T., Fontanella, S., & Adachi, K. (2017). Sparse Exploratory Factor Analysis. Psychometrika, (2011). 
Some of these techniques are already implemented in R packages (e.g., regSEM or fanc), so they are quite accessible. 
