0
$\begingroup$

I am struggling a little bit with PCA. I understand that standardization is an important part of the algorithm but I do not understand which elements should be standardized. Let's say I have a 10x100 matrix X where the 10 rows are the samples and the 100 columns are the features. Each sample is a RGB image considered as an array (my real dataset has 1087 samples, each one with 154587 features).

Should I standardize each feature or each sample? What if I do not take into account the rows and the columns and I simply do this:

X_std = (X - X.mean()) / X.var()

I can't figure out the reason why I should standardize with respect to the feature, to the sample or to the entire dataset. What I know is that the standard scaler from sklearn by default runs standardization feature-wise making each feature zero mean and unit variance.

Thank you for your help

$\endgroup$
3
  • 1
    $\begingroup$ You can do neither, one, both, or either (in either order--the order matters): it depends on what your data mean and what your objectives are. $\endgroup$
    – whuber
    Dec 28, 2018 at 23:34
  • $\begingroup$ my data are simply rgb images treated as arrays and I do not understand the meaning of standardizing directly the entire dataset with the code above versus a feature-wise or sample-wise standardization. $\endgroup$
    – matteof93
    Dec 28, 2018 at 23:40
  • $\begingroup$ Would it be better to standardize sample-wise? Might be duplicate to this question? $\endgroup$ Jan 1, 2019 at 8:51

1 Answer 1

0
$\begingroup$

You should be standardizing the dataset separated by the features. In other words, if you have a sample space X (n x d) with n samples and d features per sample, you want to view each feature as being independently distributed (in the most simple case).

So, standardize each n-length feature vector X[:, i] where i = 1,...,d

Doing this will make sure that each feature distribution is judged by the model according to the same mean (0) and variance (1). If you were to standardize each sample X[i] where i = 1,...,n, this wouldn't produce anything significant because the features will be likely coming from different distributions. This can be bad if certain features have different number ranges.

For example, if f1 is in a range of (-2, 4) and f2 is in a range of (-10000, 10000), standardizing at a sample level wouldn't make sense because all of the low-range feature values would be compressed into a tiny subspace due to the larger variance caused by higher range features.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.