Does it make sense to apply PCA or Z-Score to any dataset?

Question

Suppose we have a given dataset whose variables represent different things. For instance, one of them could represent the time a user spends on the phone while another one can represent the continent the user lives in (with a number, for example 1 for America and 2 for Europe, etc.).

Does it make sense to use PCA or normalize these datasets? I tend to see that people usually normalize their datasets before using them to train a machine learning model, or even use PCA to reduce the dimensionality of the problem. Would these approaches be correct in a dataset where the variables represent totally different things (i.e. they are not all samples of a signal during time or coordinates in the space, they are things that have very different natures)?

Answer 1

It is perfectly fine to use PCA on a dataset with well defined continuous features with vastly different scales as long as you standardize all your variables to put them back on the same scale (note that centering is a necessity for PCA as well).

The further you move away from continuous data, the more problematic it becomes. For example, PCA (and standardization) on a multinomial feature coded as numbers like numbers for continents is total nonesense.

There is a method along the lines of PCA that is meant to deal with mixed data called FAMD : Factor Analysis of Mixed Data.

See this link :

Factor Analysis of Mixed Data

Answer 2

Let's first talk about PCA use cases -

• Reducing dimensionality of the dataset - since with high dimensionality you can overfit easily, and moreover it might need more compute power.

• Removing Noise from dataset up to certain extent.

If you want any of the above from your dataset you can go about doing PCA.

Standardizing your dataset is very important. Since Classifier/Regressors like linear/logisitic can have huge impact if features are on different scales. With this in mind we usually only standardize Continuous Values/ Numbers but if in your case there are categorical features you might have to Label Encode/One-hot Encode them ( One hot encoder is better , but at the cost of increased dimensionality). The problem is that if you don't standardize ( use Z-scores ) then you usually end up with noisy weights in LR and hence the model performs poorly or it just takes longer time for the model to achieve good performance which it could have done faster had the features been standardized.