0
$\begingroup$

I am struggling a little bit with PCA. I understand that standardization is an important part of the algorithm but I do not understand which elements should be standardized. Let's say I have a 10x100 matrix X where the 10 rows are the samples and the 100 columns are the features. Each sample is a RGB image considered as an array (my real dataset has 1087 samples, each one with 154587 features).

Should I standardize each feature or each sample? What if I do not take into account the rows and the columns and I simply do this:

X_std = (X - X.mean()) / X.var()

I can't figure out the reason why I should standardize with respect to the feature, to the sample or to the entire dataset. What I know is that the standard scaler from sklearn by default runs standardization feature-wise making each feature zero mean and unit variance.

Thank you for your help

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ You can do neither, one, both, or either (in either order--the order matters): it depends on what your data mean and what your objectives are. $\endgroup$
    – whuber ♦
    Dec 28, 2018 at 23:34
  • $\begingroup$ my data are simply rgb images treated as arrays and I do not understand the meaning of standardizing directly the entire dataset with the code above versus a feature-wise or sample-wise standardization. $\endgroup$
    – matteof93
    Dec 28, 2018 at 23:40
  • $\begingroup$ Would it be better to standardize sample-wise? Might be duplicate to this question? $\endgroup$
    – Lerner Zhang
    Jan 1, 2019 at 8:51
  • $\begingroup$ for image processing reasons you might first standardise within the image, to handle lighting variation: the assumption being that each image has "same" range of colours, but only the lighting is changing. see eg en.wikipedia.org/wiki/Color_constancy#Retinex_theory $\endgroup$
    – seanv507
    Jun 16 at 16:18

1 Answer 1

Reset to default
0
$\begingroup$

You should be standardizing the dataset separated by the features. In other words, if you have a sample space X (n x d) with n samples and d features per sample, you want to view each feature as being independently distributed (in the most simple case).

So, standardize each n-length feature vector X[:, i] where i = 1,...,d

Doing this will make sure that each feature distribution is judged by the model according to the same mean (0) and variance (1). If you were to standardize each sample X[i] where i = 1,...,n, this wouldn't produce anything significant because the features will be likely coming from different distributions. This can be bad if certain features have different number ranges.

For example, if f1 is in a range of (-2, 4) and f2 is in a range of (-10000, 10000), standardizing at a sample level wouldn't make sense because all of the low-range feature values would be compressed into a tiny subspace due to the larger variance caused by higher range features.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Sorry, but is the independent part really needed? If anything, when doing PCA don't you somewhat want to see the correlation, the direction of the PA? $\endgroup$
    – Preston Lui
    Oct 5, 2022 at 12:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.