Take 20 random points in a 10,000-dimensional space with each coordinate iid from $\mathcal N(0,1)$. Split them into 10 pairs ("couples") and add the average of each pair ("a child") to the dataset. Then do PCA on the resulting 30 points and plot PC1 vs PC2.

A remarkable thing happens: each "family" forms a triplet of points that are all close together. Of course every child is closer to each of its parents in the original 10,000-dimensional space so one could expect it to be close to the parents also in the PCA space. However, in the PCA space each pair of parents is close together as well, even though in the original space they are just random points!

How do children manage to pull parents together in the PCA projection?

$\quad\quad\quad\quad$enter image description here

One might worry that this is somehow influenced by the fact that the children have lower norm than the parents. This does not seem to matter: if I produce the children as $(x+y)/\sqrt{2}$ where $x$ and $y$ are parental points, then they will have on average the same norm as the parents. But I still observe qualitatively the same phenomenon in the PCA space:

$\quad\quad\quad\quad$enter image description here

This question is using a toy data set but it is motivated by what I observed in a real-world data set from a genome-wide association study (GWAS) where dimensions are single-nucleotide polymorphisms (SNP). This data set contained mother-father-child trios.


%matplotlib notebook

import numpy as np
import matplotlib.pyplot as plt

def generate_families(n = 10, p = 10000, divide_by = 2):
    X1 = np.random.randn(n,p)    # mothers
    X2 = np.random.randn(n,p)    # fathers
    X3 = (X1+X2)/divide_by       # children
    X = []
    for i in range(X1.shape[0]):
        X.extend((X1[i], X2[i], X3[i]))
    X = np.array(X)

    X = X - np.mean(X, axis=0)
    U,s,V = np.linalg.svd(X, full_matrices=False)
    X = U @ np.diag(s)
    return X

n = 10
X = generate_families(n, divide_by = 2)
for i in range(n):
    plt.scatter(X[i*3:(i+1)*3,0], X[i*3:(i+1)*3,1])

X = generate_families(n, divide_by = np.sqrt(2))
for i in range(n):
    plt.scatter(X[i*3:(i+1)*3,0], X[i*3:(i+1)*3,1])
  • 1
    $\begingroup$ In so high dimension all data points of a random unrorrelated data are located in the corners of the space and the distances between the points are almost the same. If you select a point and tie it with another one of the points by way of creating a half-way point (average) between them you thus have created a cluster: you've introduced distances distinctly smaller than the distance mentioned earlier. $\endgroup$ – ttnphns Nov 11 '18 at 16:52
  • 1
    $\begingroup$ Yes, I understand that the 20 original points are all more or less equidistant from each other. And it's clear that the children are closer to their parents than any two parents are to each other. What I still don't get though, is why the parents become close in the PCA projection... $\endgroup$ – amoeba Nov 11 '18 at 17:58
  • $\begingroup$ Did you try to project on any two random dimensions? What did you get? $\endgroup$ – ttnphns Nov 11 '18 at 18:35
  • 1
    $\begingroup$ My intuition would be this: the triplets of the points are projected as piles almost perpendicular to the PC1-2 subspace. This is how the position of this plane is defined to maximize variance. You see, you've got multimodal data with modes mostly away from the centre (because points are are all periferal in 10K dim), such a cloud, like a dumbbell, will tend to pull the main PCs so that these pierce the heavy regions, and therefore perpendicular to the triplets. $\endgroup$ – ttnphns Nov 11 '18 at 20:13
  • 1
    $\begingroup$ The finding, btw is instructive to the problem that PCA (PCoA) is not very good as a MDS because it projects points and does not model distances directly. An iterative MDS would expected to produce those "clusters" to a much lesser degree. $\endgroup$ – ttnphns Nov 11 '18 at 20:49

During the discussion with @ttnphns in the comments above, I realized that the same phenomenon can be observed with many fewer than 10 families. Three families (n=3 in my code snippet) appear roughly in the corners of an equilateral triangle. In fact, it is enough to consider only two families (n=2): they end up separated along PC1, with each family projected roughly onto one point.

The case of two families can be visualized directly. The original four points in the 10,000-dimensional space are nearly orthogonal and reside in a 4-dimensional subspace. So they form a 4-simplex. After centering, they will form a regular tetrahedron which is a shape in 3D. Here is how it looks like:

*enter image description here*

Before the children are added, PC1 can point anywhere; there is no preferred direction. However, after two children are positioned in the centers of two opposite edges, PC1 will go right through them! This arrangement of six points was described by @ttnphns as a "dumbbell":

such a cloud, like a dumbbell, will tend to pull the main PCs so that these pierce the heavy regions

Note that the opposite edges of a regular tetrahedron are orthogonal to each other and are also orthogonal to the line connecting their centers. This means that each family will be projected to one single point on PC1.

Perhaps even less intuitively, if the two children are scaled by the $\sqrt{2}$ factor to give them the same norm as the parents have, then they will "stick out" of the tetrahedron, resulting in PC1 projection with both parents collapsed together and child being further apart. This can be seen in the second figure in my question: each family has its parents really close on the PC1/PC2 plane (EVEN THOUGH THEY ARE UNRELATED!), and their child is a bit further apart.

| cite | improve this answer | |
  • 3
    $\begingroup$ Excellent visualization! Mom1-Child1-Dad1 is one disc or pancake, and Mom2-Child2-Dad2 is the other one, of the bimodal cloud. It attracts the PC1, in order to maximize the variance of the projection, to pierce both "families" orthogonally to their mom-child-dad lines. Consequently, each family projects into one point (a child, in this instance) and we have the two families as two very tight inside, distant from each other clusters in the projection. $\endgroup$ – ttnphns Nov 12 '18 at 15:24
  • 1
    $\begingroup$ What program did you use to draw the pic? $\endgroup$ – ttnphns Nov 12 '18 at 15:24
  • 5
    $\begingroup$ Whiteboard, whiteboard markers, and a smartphone camera :-) $\endgroup$ – amoeba Nov 12 '18 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.