# Dimensionality reduction: include labels?

Lets say I have something like the iris dataset, the columns are petal length, petal width, sepal length, sepal width and species.

I want to reduce this to 2 dimensions, create 5 separate 2D plots, each with points coloured by value of each of the 5 columns. Then I want to compare across graphs to identify structure and relationships between variables. For example, are the points with large petal length all clustered together in the same region? Are those same points also the ones that belong to a single species?

My question: when reducing down to 2 dimensions (using whichever technique, PCA, SNE, t-SNE), is it appropriate to also include the labels (i.e. species) in that model?

The choice would be between running PCA on:

1. petal length, petal width, sepal length, sepal width and species, or

2. petal length, petal width, sepal length, sepal width

Species will be one of my resulting 5 plots, but are there any issues with including it in the initial PCA model? Which choice is more defensible?

On the one hand it makes sense not to include because the other 4 variables all capture something similar: physical attributes of the flowers. But on the other hand I would expect species to be highly related to those attributes, even if its not a physically measured attribute like the others

Ive seen it both ways in research papers, but more common to leave the label out of the initial model, and only bring it in when you create the colour plots

EDIT: in my actual dataset all of the variables (including the "labels") are continuous and not categorical like species

heres an example of what I want using the mtcars data:

I took all 7 continuous/ordinal variables and ran them through PCA to reduce to 2 dimensions. Then I created separate plots for each variable. The purpose of this visualization is to compare across plots. For example, maybe I'm interested in seeing which car performance variables are related to which physical measures of the car, and to say something like cyl and mpg seem to be inversely related as high cylinder count seems to be associated with low mpg

My question is whether all 7 variables should be included in the original PCA.

Notice that some of those variables (e.g. cyl, drat, wt) are related to physical aspects of the car, whereas others (e.g. mpg, qsec) are more about performance. Even though physical attributes and performance are related, they slightly different categories. So maybe it is more correct to run PCA only on the physical variables, but then to colour by all 7?

Given I'm primarily interested in how performance attributes are related to physical attributes, would it make more sense to PCA on the physical measures only (as the data points would then represent a single category - something about the physical makeup of the cars)? Are there any issues with including both performance and physical variables in the PCA, when conceptually the performance measures are what I'm interested in seeing relationships with and would therefore function like my "labels"

If you include the labels, you wouldn't be able to use it in your pipeline to make predictions, because there would be no labels in the future data (otherwise, you wouldn't need to predict them). When including them, you leak information from the labels, so if you tested it on your train/dev/test samples, the performance metrics would be inflated and incorrect.

• understood, but Im not interested in making any predictions. I only want to plot the data to visualize relationships between variables. Is that still an issue in this situation? – Simon Nov 28 '18 at 8:51
• What for you want to make the visualizations? – Tim Nov 28 '18 at 8:57
• I want to see how specific variables are distributed across the lower dimensional space, and whether they tend to cluster together, and whether that clustering is similar to the clustering of other variables in my data. so the goal is to compare distribution across plots, similar to what you might do with something like a self-organizing map – Simon Nov 28 '18 at 9:29
• @Simon but cluster together in terms of labels? If yes, then the better approach would be not to include labels, but after reducing dimensionality check how do they relate to the labels. – Tim Nov 28 '18 at 9:32
• I edited my question to clarify what I'm looking for. But what would be the issues with including "labels" in the PCA given they should be related to the other variables? I can understand the problem with that in a prediction case (as you're essentially training with your test data), but Im not sure I understand the issue when it comes to pure visualization – Simon Nov 28 '18 at 19:27

Whether you include species or not depends on your needs and affects your interpretation. As Tim indicated it is nonsensical for a predictive model but let's consider what it means for exploratory data analysis.

If you include species as a variable in the data reduction step, then it appears in the eigenvectors with its own weight. This means the resulting scores include covariance of species with each of the other variables. An issue is that species is categorical, not ordinal so the numeric relationships may be very difficult to i interpret.

Another alternative is to split the species into a series of binary variables for each class, so there is no longer numeric issues with arbitrary ordering, but now you have species spread over several variables and that can complicate interpreting the eigenvectors too.

It is more usual to use label variable to colour by group for visual display. This allows us to see how the categorical classes are distributed across the pcs and is simpler to interpret than untangling the mathematical consequences of arbitrary ordering of a categorical variable's value.

• So I used the iris dataset as an example (probably a bad example now that I think about it), but in reality my "labels" are continuous/numeric. Would there be any concerns specific to coloring by continuous variable if they're also included in the PCA? – Simon Nov 28 '18 at 9:26
• Could you update your question with a clearer explanation of what your thinking is and what your concerns are? Are the labels continuous (in which case label is an odd term) or ordinal multinomial? – ReneBt Nov 28 '18 at 10:04
• I updated my question, hopefully it makes a bit more sense. "Labels" are both continuous and ordinal in my case – Simon Nov 28 '18 at 19:28