This question falls into the area known as 'factorial invariance'. There are several kinds of invariance, and it's not absolutely clear which you mean.
One problem is that males and females have different levels of the indicators, even when the level of the trait being studied is the same. One example is crying as a symptom of depression. Given the same level of depression, women tend to cry more often than men. If you treat that variable equally for males and females, the female scores will be biased high. Another problem that will arise is that if there is another item that women score differently on, for the same level of depression, then you are at risk of extracting a 'gender' factor. One solution to this is to regress each variable on gender, and save the residuals. This fixes the mean for each group, for each variable, to be zero. Equivalently, for each item, subtract the male mean from the female mean, and all groups will have the same mean.
In factorial invariance, we consider several levels of this.
- Configural invariance: same factor structure holds (This is what @Md Azimul Haque is discussing)
- Weak factorial invariance: across groups, corresponding factor loadings are equivalent
- Strong factorial variable intercepts are equivalent.
- Strict factorial invariance: residual variances are equal.
The problem my answer deals with is (3), which is often the most serious The problem that @Md Azimul Haque deals with is (1), however the approach that they describe isn't very useful, as you can't tell if it is sampling variation or actual differences when using PCA. Factorial invariance is usually investigated using confirmatory factor analysis.
There's a book by Roger Millsap which goes into considerable detail on this.
(In addition, I don't think you should be using principal components analysis; this sounds like a factor analysis problem. But that's another question.)