PCA based filtering is used to identify and eliminate noise in data. This would basically involve computing the PCs and using the top k PCs to denoise the data. What if I know for sure that only the extremely small values in my matrix are noise? Now, a value may be small w.r.t the entire matrix but not small w.r.t a particular row/column. Can I achieve this with some transformation of the input matrix followed by a PCA based filter?
PCA depends on the scaling of your columns. If you perform a PCA on matrix $X$, then rescale each column to be norm 1 (i.e. divide by the two norm of each column), then perform a PCA on the transformed $X$, you will get different answers. I believe this is part of what 'small w.r.t. a particular row/column' is referring to in the original question.
However, the small elements in the matrix should contribute very little to a PCA/downsample/reconstruction operation. Perhaps you would be better served by determining, perhaps in a semiautomatic way, how many PCs to take in the representation. For this purpose, you might want to look at the scaling of the eigenvalues you get from the SVD, and look for the 'knee' in the scree plot, or if you know something about the small noise, you can rely on the distribution of eigenvalues of the Gramian matrix. For normally distributed noise, with equal variances, they should follow a Marchenko-Pastur Distribution, up to scaling. This will give you the limits of the eigenvalues you expect to see in the pure noise situation. Basing your PC cutoff on that limit may be fruitful.
Sorry this is somewhat vague, I do not think I fully understand what technically is desired from the OP.