I have calculated Hotelling's T2 statistic for detection of outliers in PCA analysis in Matlab. However, I am unsure as to whether or not it is a robust approach to remove these outliers? The output will be used in a cluster analysis and I am wondering if I remove the outliers, am I fundamentally changing the outcome of the cluster analysis in a very subjective way? ie. the outliers could be the thing that makes a specific data point belong to a specific group. So, my question is: Is it generally the rule that outliers are removed from PCA? I see a lot online about identifying them, but not so much about what to do with them afterwards.
As a very general rule, the proper treatment of outliers depend on the analysis purpose - if you're looking for large-scale tendencies, they often better be removed, but sometimes your goal might be actually finding the non-typical data points. Now, going into details... PCA is a method based on correlation/covariance matrix, which can be - at least in theory - very sensitive to outliers. One way to get around this problem is to use a robust correlation matrix - see e.g. https://cran.r-project.org/web/packages/robust/robust.pdf or, if you use Matlab, maybe https://sourceforge.net/projects/robustcorrtool/ will be better. Clustering can also serve as a outlier detection technique, but if you want to identify a few groups of similar points in the dataset, I'd suggest removing the outliers since - again - they can affect the workings of some clustering algorithms (like k-means, which is based on within-cluster variance) and make the results harder to interpret. We usually like the elements of a cluster to be rather homogeneous, and the very definition of outlier contradicts similarity to other cluster members... EDIT: sorry for not making myself clear - if you base PCA on robust covariance, there's no need to remove anything. When using 'ordinary' correlation, you might compare the results with and without outliers and see if they make any real difference.