When you do PCA (or any dimensionality reduction), what is "the number of dimensions"?

Question

Fundamental question

When you do PCA (or any dimensionality reduction), what is "the number of dimensions"? I always thought that the thing you measure (ie, the variable) is the number of dimensions: eg, if you measure the length, width, height of a box, that's 3 dimensions (3 variables); if you measure the abundance of 10,000 genes in 200 cells, that's 10,000 dimensions (not 200 dimensions).

More specific question

In regard to image 1 (below), what is the "correct" interpretation of number of dimensions (before PCA); is it the number of cells (200), or the number of genes (10,000)?

Note: I think it's possible to use either the number of cells or the number of genes as the number of dimensions, with obviously different interpretations. Additionally, there are some other good discussions of PCA on Cross Validated; however, my question is a little bit different: I'm really hoping for a response regarding my confusion after watching this video on PCA. Here's a brief explanation of my confusion.

The narrator is trying to explain PCA in context of this experiment (Image 1, below):

This graph was drawn from single-cell RNA-seq. There were about 10,000 transcribed genes in each cell.

Each dot represents a single-cell and its transcription profile. The general idea is that cells with similar transcription should cluster.

As I thought I understood PCA, in this experiment, the genes are the "dimensions" and the cells are the observations; ie, if there are 10,000 genes, there are 10,000 dimensions. This understanding seems to coincide with a different example (see references) using the Iris data set (Image 2, below); as you can see, the number of dimensions is the number of features of the flowers that were measured.

However, in the video, the narrator goes on to describe the number of dimensions as the number of cells for which the experiment measured the gene abundance (see Images 3 and 4, below):

Q: In regard to the experiment in Image 1 where "each dot represents a single cell", was the number of dimensions (before PCA) the number of cells or the number of genes?

References:

Link to video: https://www.youtube.com/watch?v=_UVHneBUBW0

For link to reference on the Iris data, google "Principal Component Analysis in 3 Simple Steps Sebastian Raschka" (I don't have enough reputation on this site to include more links in this question).

Answer 1

(I cannot watch the video right now so this answer is to some extent a guess of what is meant)

First of all, yes we mostly talk about variates spanning the dimensions. However, it is also possible to take the opposite view (this is sometimes calles R-mode vs. Q-mode analysis).

Let me take a detour to cluster analysis to illustrate this: Cluster analysis with variates = genes = dimensions will look for groups of cases that have similar gene expression patterns. On the other hand, you can also take a "transposed" view and ask for groups of genes which are expressed similarly for the same cells. Genes and cells have changed their role compared to the first approach. For some types of data, you may get similar groupings both ways (see e.g. our paper on using this for spectroscopic data: A. Bonifacio, C. Beleites and V. Sergo: Application of R-mode analysis to Raman maps: a different way of looking at vibrational hyperspectral data, AnalBioanalChem, 407, 4 (2015) 1089–1095. DOI 10.1007/s00216-014-8321-7) whereas for other types of data both ways of looking at the data are interesting in themselves (e.g. for genetic data). In the latter case, you can use a heatmap giving both ways of clustering.

Now for PCA, the fun fact is that up to some decisions of standardization (row vs. columns for centering and possibly scaling) you'll arrive at the same solution both ways - just scores and loadings will change their role. (see e.g. https://stats.stackexchange.com/a/147983/4598 and Why PCA of data by means of SVD of the data? for more details)

Is the number of dimension the number of cells or the number of genes?

IMHO this is rather ambiguious and as explained above depends on the view of the data you take (i.e. the question you ask/the application at hand).

For PCA, there's the additional ambiguity that "dimensions" is sometimes also used refering to the rank of the data matrix. The rank cannot be more than the smaller of number of rows and number of columns and it is also the maximum number of principal components for that data matrix and thus the number of dimensions of the resulting rotated coordinate system (before reducing dimensions by truncating this coordinate system).

In your example of 200 cells and 10⁴ genes the PC will at most span 200 dimensions, regardless of whether cells or genes were considered the variates by the mode of the data analysis.

Answer 2

An extensive discussion is already provided here in the answer by cbeleites, and under similar questions (PCA and Correspondence analysis in their relation to Biplot), so I'll just comment briefly on the specific video.

As the narrator never mentions "scores" or "loadings" explicitly throughout the video, and the term "dimensions" in PCA is already ambiguous, technically there were no mistakes. However, I agree with you that his presentation is confusing: the first part states that the dimensionality along cells is to be reduced (200 cells -> 2 PCs), and the second part actually focuses on reducing dimensionality along genes (10,000 genes -> 2 PCs). I'd say there's many better and still accessible introductions to PCA, with consistent presentation and actual terminology E.g.: http://webspace.ship.edu/pgmarr/Geo441/Lectures/Lec%2017%20-%20Principal%20Component%20Analysis.pdf .

Answer 3

The definition of a vector space is quite general and there are numerous ways to represent data as vectors in a vector space.

From my cursory examination, they may be doing the following?

• Let $i = 1, \ldots, m$ index the gene.
• Let $j = 1, \ldots, n$ index the cell.
• Let $x_{i,j}$ denote the gene expression level of gene $i$ in cell $j$.

We then have a matrix of data $X$. You can run PCA on either $X$ or the transpose $X^T$.

Treat columns of X as vectors (what you're naturally thinking)

You're thinking we can construct a vector for each cell $j$ as:

$$ \mathbf{y}_j = \begin{bmatrix} x_{1,j} \\ x_{2,j} \\ \ldots \\ x_{m,j} \end{bmatrix}$$

That is, each vector $\mathbf{y}_j$ shows the gene expression levels of cell $j$ (and each index is a different gene).

Treat rows of X as vectors (what they appear to be doing in Image 3 and 4?)

We could also form vectors using the columns of $X$

$$ \mathbf{z}_i = \begin{bmatrix} x_{i, 1} \\ x_{i, 2} \\ \dots \\ x_{i, n} \end{bmatrix} $$ That is, each vector $\mathbf{z}_i$ shows the gene expression levels of gene $i$ (and each cell is a different index of the vector).

Onward to PCA

Once you have a bunch of vectors, you can always conduct PCA to find an alternative basis for that space.

A subject area specific interpretation of that basis of course will depend on what your various vectors represent.

Answer 4

I don't like these videos. They only make understanding PCA more difficult by bringing in irrelevant details. Also, they're long and wordy.

The idea of PCA is very simple when it comes to applications. You have a several series of data, call them variables. Say you have N variables (series) $x_1(t),x_2(t),\dots,x_N(t)$.

Sometimes there are a few underlying factors that drive all these series. Let's say there are M factors $f_1(t),f_2(t),\dots,f_M(t)$, and you suspect (or know for sure) that they drive the variables: $$x_1(t)=c_{11}f_1(t)+\dots+c_{1M}f_M(t)\\\dots\\ x_N(t)=c_{N1}f_1(t)+\dots+c_{NM}f_M(t)$$

So, you're interested to extract the factor values $f_j(t)$ and the coefficients $c_{ij}$. So, PCA is one way of accomplishing this. In fact, if you don't know what is the exact number of components M, it can help you find that out too.