I am experimenting PCA to detect outliers based on the reconstruction error.
What I do: I start with a 6 dimensions dataset and reduce it to 5 dimensions. Then, I reconstruct the initial dataset and end up with a reconstruction error for each observation. The observations associated with the highest reconstruction error will be defined as outliers.
The variables in my initial dataset have very low linear correlation (the maximum for two of them is 0.4).
Question 1: does it make sense to detect outliers with PCA if I have very low linear correlations between my variables?
I tried to draw an example in 2D to illustrate my concerns. The two variables I simulated have very low correlation (=0.25). The PCA would consist in reducing the 2 dimensions to 1 dimension only by projecting the points into the direction given by the eigenvector with the highest magnitude (eigenvalue). Say the green eigenvector has the highest magnitude. Then the highest projection errors will be associated with the points in the orange circles.
The problem here is that I am missing a large number of outliers that are situated in other locations. Moreover, the method seems not well adapted with this data shape
Question 2: any suggestion/idea on how I could identify whether I am in such case?