PCA summarises the covariance structure of the training set data and will therefore reflect all variance present on that set.Will PCA detect all anomalous behaviour? No method will, but it has its strengths, but its biggest one is in pattern handling, and identifying unusual patterns which alas is not what you are describing. You appear to be talking about rare events confined to one variable, but there is still a lot PCA can do
Two issues arise for outliers
1)in training/calibration. Are any samples presenting variance that is not well represented throughout the dataset? If only one sample presents a behaviour then your PCA model will not describe that behaviour reliably. Many methods exist to identify such issues, including Hotelling's $T^2$, distance to model, leverage, residuals (both of the latter 2 can be used sample or variable wise) . It is a hot topic, and no answer is universally applicable. To me if any variation is not well described the samples should be removed or the experiment redesigned as otherwise they create an unreliable element in your model that will behave unpredictably in new datasets as you have neither good understanding of its variance nor its covariance with everything else.
2) in the test /validation /application. All PCA models should build in sanity checks to determine if there are significant residual variation that the model does not explain in new data so that you can estimate how well the model describes the sample. If there is a lot of unexplained variance then you are extrapolating and should proceed with caution.
.
Our consideration is that the PCA will neglect this feature and when >we will reduce the number of columns after the PCA (say we take >95% of data) the anomaly will "disappear".
Not if you use PCA correctly, if you look beyond your basic eigenvectors at the metrics mentioned above you would see any such behaviour. Where few samples or variable are anomalous and causing an outsize influence on the model these often are detectable in leverage , while residuals are good for ensuring the variation of specific samples or variables has been accounted for by your chosen number of PCs.
If the problem is that you are looking at a rare event that you specifically want the model to handle, then the problem is whether you have powered your study sufficiently well to get a reliable estimate of its behaviour, not a problem with PCA itself. There are also things can be done with Design Of Experiment to ensure that maximal relevant variance is captured with an efficient dataset.
Is using PCA for finding anomalies discouraged? or we are missing something?
I would say that using PCA for finding anomalies should be encouraged, but the full range of tools need to be explored to look for different types of anomalies. Anomalies however may reflect a study design inadequate for the variation of interest.