# How to visualize the true dimensionality of the data?

I have a dataset that's nominally 16-dimensional. I have about 100 samples in one case and about 20,000 in another. Based on various exploratory analyses I've conducted using PCA and heat maps, I'm convinced that the true dimensionality (i.e. the number of dimensions needed to capture most of the "signal") is around 4. I want to create a slide to that effect for a presentation. The "conventional wisdom" about this data, which I'm looking to disprove, is that the true dimensionality is one or two.

What's a good, simple visualization for showing the true dimensionality of a dataset? Preferably it should be understandable to people who have some background in statistics but are not "real" statisticians.

-
Define "true", please. –  cardinal Mar 9 '11 at 18:04
@cardinal: It's unavoidably subjective. I want the amount of dimensions that capture "most" of the variance. –  dsimcha Mar 9 '11 at 18:08
that's fine. I just wanted to know for sure what you were going for. There are interesting examples in multivariate statistics where, e.g., if you want to find the best number of terms to keep in a least-squares sense for PCA, you should pick a smaller rank than the true rank, i.e., the rank of the actual underlying structure from which the data were generated. –  cardinal Mar 9 '11 at 18:13
I would find a specific case where using 2-dimensions leads to a bad answer, while using 4-dimensions leads to a good answer. This will help in getting your point across - people understand examples. Add this to some of the more general stuff given below - "eblow" in the scree plot, etc. –  probabilityislogic Mar 10 '11 at 23:21
the "examples" would be those cases which have large scores on the third and forth components –  probabilityislogic Mar 10 '11 at 23:22

A standard approach would be to do PCA and then show a scree plot, which you ought to be able to get that out of any software you might choose. A little tinkering and you could make it more interpretable for your particular audience if necessary. Sometimes they can be convincing, but often they're ambiguous and there'a always room to quibble about how to read them so a scree plot may (edit: not!) be ideal. Worth a look though.

-
@JMS, (+1) especially for the note of caution on scree plots. I would call them "standard", but also "a bad idea" (generally speaking). Here is a cautionary tale and also a way to pick the rank more safely, especially if MSE is your metric. This also provides an example, if I recall, where the true rank is the wrong one to pick for minimizing MSE. –  cardinal Mar 9 '11 at 18:29
@cardinal (+1) Thx for the link to Owen & Perry's article. –  chl Mar 9 '11 at 18:33
I apparently reinvented scree plots at one point but didn't realize they were called anything. Thanks for reminding me of them and letting me know that they are "standard". Since my goal is to produce a nice visualization, simple, standard and good enough wins out over complicated but more optimal. –  dsimcha Mar 9 '11 at 19:08
@dsimcha, I did a quick Google search and there is apparently an R package that implements the approach in the link I provided. There is a nice paper from about 20 years ago (maybe 30), I believe in The American Statistician, which clearly demonstrates the flaws of inference using "standard" scree plots. I'll see if I can recall the title and will post a link or reference. While standard, but (quite) flawed, is an appealing approach, especially under time constraints, I would gently caution against such a mindset. Cheers. –  cardinal Mar 9 '11 at 20:56
@JMS There is a paper by Zwick & Velicer (1986) that argues for using parallel analysis (resampling eigenvalues to test whether observed values were greater than expected by chance) or their MAP criterion over scree plots or the root-one rule. In their simulations the first two methods heavily outperform the others (e.g. 92% vs. 22% accuracy) in identifying the true # of components. The R package psych implements both with accompanying scree-like graphics (see fa.parallel() and VSS()). Paper is "Comparison of five rules for determining the number of components to retain." –  lockedoff Mar 10 '11 at 21:52

One way to visualize this would be as follows:

1. Perform a PCA on the data.
2. Let $V$ be the vector space spanned by the first two principal component vectors, and let $V^\top$ be the complement.
3. Decompose each vector $x_i$ in your data set as the sum of an element in $V$ plus a remainder term (which is in $V^\top$). Write this as $x_i = v_i + c_i$. (this should be easy using the results of the PCA.)
4. Create a scatter plot of $||c_i||$ versus $||v_i||$.

If the data is truly $\le 2$ dimensional, the plot should look like a flat line.

In Matlab (ducking from all the shoes being thrown):

lat_d = 2;   %the latent dimension of the generating process
vis_d = 16;  %manifest dimension
n = 10000;   %number of samples
x = randn(n,lat_d) * randn(lat_d,vis_d) + 0.1 * randn(n,vis_d); %add some noise
xmu = mean(x,1);
xc = bsxfun(@minus,x,xmu);    %Matlab syntax for element recycling: ugly, weird.
[U,S,V] = svd(xc);  %this will be slow;
prev = U(:,1:2) * S(1:2,1:2);
prec = U(:,3:end) * S(3:end,3:end);
normv = sqrt(sum(prev .^2,2));
normc = sqrt(sum(prec .^2,2));
scatter(normv,normc);
axis equal;  %to illlustrate the differences in scaling, make axis 'square'


This generates the following scatter plot:

If you change lat_d to 4, the line is less flat.

-
@shabbychef, this seems to presuppose that (a) the true structure is a linear manifold and (b) the high variance directions are the important ones. –  cardinal Mar 9 '11 at 18:09
@cardinal You may be confusing assumption and effect. Curvature in the manifold will be manifest as curvature in these scatterplots and the magnitude of curvature will be revealed in the magnitude of vertical variation in the plots. In effect, shabbychef is viewing the data as residing approximately within a tubular neighborhood of a low-dimensional linear space. That imposes no restrictions whatsoever on the shape of the data. –  whuber Mar 9 '11 at 18:57
@shabbyshef +1 for the decomposition idea, and of course the mentioning of the shoes :) –  mpiktas Mar 10 '11 at 7:57
@whuber, @shabbychef, I was probably a little hasty with my initial comment, especially since I was away from a computer at the time. I think I also initially misread shabby's description and I couldn't view the plots. These make a nice way to augment a standard pairs plot. –  cardinal Mar 10 '11 at 13:48
@cardinal Here's a 4D example generalizing your cylinder: i.imgur.com/9eF8N.png . The model is $(s,t)\to(\cos(2\pi t),\sin(2\pi t),2s/5,t^2)$ plus normal(0,.05) error added independently to all four components. The image is a scatterplot matrix of 300 iid draws with $s$ and $t$ iid uniform(0,1), ordered by decreasing variance. The proportions of total variance are 49.2%, 46.5%, 2.9%, 1.4%. The two bottom rows emulate @shabby's plots. They demonstrate the near 2D nature of the data while revealing the small nonlinearity I put into it. Is this what you were thinking of? –  whuber Mar 10 '11 at 21:33