Dimensionality reduction: include labels?

Question

Lets say I have something like the iris dataset, the columns are petal length, petal width, sepal length, sepal width and species.

I want to reduce this to 2 dimensions, create 5 separate 2D plots, each with points coloured by value of each of the 5 columns. Then I want to compare across graphs to identify structure and relationships between variables. For example, are the points with large petal length all clustered together in the same region? Are those same points also the ones that belong to a single species?

My question: when reducing down to 2 dimensions (using whichever technique, PCA, SNE, t-SNE), is it appropriate to also include the labels (i.e. species) in that model?

The choice would be between running PCA on:

1. petal length, petal width, sepal length, sepal width and species, or

2. petal length, petal width, sepal length, sepal width

Species will be one of my resulting 5 plots, but are there any issues with including it in the initial PCA model? Which choice is more defensible?

On the one hand it makes sense not to include because the other 4 variables all capture something similar: physical attributes of the flowers. But on the other hand I would expect species to be highly related to those attributes, even if its not a physically measured attribute like the others

Ive seen it both ways in research papers, but more common to leave the label out of the initial model, and only bring it in when you create the colour plots

EDIT: in my actual dataset all of the variables (including the "labels") are continuous and not categorical like species

heres an example of what I want using the mtcars data:

I took all 7 continuous/ordinal variables and ran them through PCA to reduce to 2 dimensions. Then I created separate plots for each variable. The purpose of this visualization is to compare across plots. For example, maybe I'm interested in seeing which car performance variables are related to which physical measures of the car, and to say something like cyl and mpg seem to be inversely related as high cylinder count seems to be associated with low mpg

My question is whether all 7 variables should be included in the original PCA.

Notice that some of those variables (e.g. cyl, drat, wt) are related to physical aspects of the car, whereas others (e.g. mpg, qsec) are more about performance. Even though physical attributes and performance are related, they slightly different categories. So maybe it is more correct to run PCA only on the physical variables, but then to colour by all 7?

Given I'm primarily interested in how performance attributes are related to physical attributes, would it make more sense to PCA on the physical measures only (as the data points would then represent a single category - something about the physical makeup of the cars)? Are there any issues with including both performance and physical variables in the PCA, when conceptually the performance measures are what I'm interested in seeing relationships with and would therefore function like my "labels"

Answer 1

If you include the labels, you wouldn't be able to use it in your pipeline to make predictions, because there would be no labels in the future data (otherwise, you wouldn't need to predict them). When including them, you leak information from the labels, so if you tested it on your train/dev/test samples, the performance metrics would be inflated and incorrect.

Answer 2

Whether you include species or not depends on your needs and affects your interpretation. As Tim indicated it is nonsensical for a predictive model but let's consider what it means for exploratory data analysis.

If you include species as a variable in the data reduction step, then it appears in the eigenvectors with its own weight. This means the resulting scores include covariance of species with each of the other variables. An issue is that species is categorical, not ordinal so the numeric relationships may be very difficult to i interpret.

Another alternative is to split the species into a series of binary variables for each class, so there is no longer numeric issues with arbitrary ordering, but now you have species spread over several variables and that can complicate interpreting the eigenvectors too.

It is more usual to use label variable to colour by group for visual display. This allows us to see how the categorical classes are distributed across the pcs and is simpler to interpret than untangling the mathematical consequences of arbitrary ordering of a categorical variable's value.