The PDFs (distributions) of the underlying variables or inputs to your PCA are really the key to how you address your question. The most important consideration is scaling which means that 1) they are "continuously" distributed, and 2) that they have been standardized to a mean of zero and std dev of one prior to the PCA. Of course, standardizing is only needed for OLS PCA since it's not a scale-invariant technique and would result in factors dominated by those variables with the largest std deviations. Maximum likelihood PCA is scale-invariant. That said, PCA with categorical or dummy variables is not a good idea. A separate class of dimension reduction techniques for categorical information (e.g., correspondence analysis) has been developed for data of this type.
A few things are always worth reviewing about PCA. First and foremost, it is a highly subjective technique dependent on analyst choice across a myriad of possible model options and specifications. Basically, this means KISS. Next, and in the absence of rotation, PCA produces 1) a mathematically unique solution, 2) orthogonal (right angled, uncorrelated), linear components, 3) Gaussian-like residuals, and 4) the first factor is a kind of "junk" factor (e.g., see Morris Holbrook's -- Columbia Marketing -- papers on FA from the 80s) where many of the input variables will load highly on the first dimension or PC.
Given your description, one immediate option would be to try rotating the PCA solution. There are many ways of doing this, but start with a plain vanilla, orthogonal rotation such as varimax. This would eliminate the uniqueness of the solution, but could provide a better fit in terms of the loadings (or correlations) of the inputs relative to the solution. The usual goal of rotation is to find "simple structure" across the dimensions. The objective of rotation to simple structure is to adjust the loadings in such a way that the inputs load on one dimension and are zero otherwise. Note that rotation does not change the component eigenvalues or "variance explained" across the dimensions. Next, you've indicated that there is strong correlation between the resulting components. This suggests trying an oblique rotation which would relax the constraint that the factors be orthogonal. Promax is one such approach.
Since we know nothing about the variables that you're using as inputs, it could be that taking some transformations prior to doing PCA could improve the solution in the absence of rotation since all PCA is sensitive to outliers. The goal of these transformations would be to further symmetrize any skewness in the residuals output from the untransformed model. Transformations which compress or expand the input PDFs are to be leveraged.
Based on the plethora of 20th c tricks that were developed for outlier identification and deletion, many statisticians would recommend dropping outlying values from the analysis. Personally, I strongly recommend not deleting outliers as they contain valuable information about the behaviors of the inputs, that is, in the absence of ancillary information confirming that these outlying values aren't illegal, fraudulent, erroneous or otherwise "bad" data.
One additional possible avenue is to leverage robust approaches to PCA. These methods are less sensitive to outliers or leverage probabilistic distributions that explicitly model that information. Here are some links to papers that describe them. Both papers have extensive bibliographies for deeper dives. Bear in mind that the cumulative impact from applying any or all of these nontraditional options may completely overturn the resulting solution relative to expectations based on published research using traditional PCA.
Robust Principal Component Analysis? by Candes, Li, Ma and Wright, 2009
Cauchy Principal Component Analysis, Xie and Xing, 2015
