How to visualize the true dimensionality of the data? I have a dataset that's nominally 16-dimensional.  I have about 100 samples in one case and about 20,000 in another.  Based on various exploratory analyses I've conducted using PCA and heat maps, I'm convinced that the true dimensionality (i.e. the number of dimensions needed to capture most of the "signal") is around 4.  I want to create a slide to that effect for a presentation.  The "conventional wisdom" about this data, which I'm looking to disprove, is that the true dimensionality is one or two.  
What's a good, simple visualization for showing the true dimensionality of a dataset?  Preferably it should be understandable to people who have some background in statistics but are not "real" statisticians.
 A: A standard approach would be to do PCA and then show a scree plot, which you ought to be able to get that out of any software you might choose. A little tinkering and you could make it more interpretable for your particular audience if necessary. Sometimes they can be convincing, but often they're ambiguous and there'a always room to quibble about how to read them so a scree plot may (edit: not!) be ideal. Worth a look though.
A: One way to visualize this would be as follows: 


*

*Perform a PCA on the data.

*Let $V$ be the vector space spanned by the first two principal component vectors, and let $V^\top$ be the complement.

*Decompose each vector $x_i$ in your data set as the sum of an element in $V$ plus a remainder term (which is in $V^\top$). Write this as $x_i = v_i + c_i$. (this should be easy using the results of the PCA.)

*Create a scatter plot of $||c_i||$ versus $||v_i||$. 


If the data is truly $\le 2$ dimensional, the plot should look like a flat line.
In Matlab (ducking from all the shoes being thrown): 
lat_d = 2;   %the latent dimension of the generating process
vis_d = 16;  %manifest dimension
n = 10000;   %number of samples
x = randn(n,lat_d) * randn(lat_d,vis_d) + 0.1 * randn(n,vis_d); %add some noise
xmu = mean(x,1);
xc = bsxfun(@minus,x,xmu);    %Matlab syntax for element recycling: ugly, weird.
[U,S,V] = svd(xc);  %this will be slow;
prev = U(:,1:2) * S(1:2,1:2);
prec = U(:,3:end) * S(3:end,3:end);
normv = sqrt(sum(prev .^2,2));
normc = sqrt(sum(prec .^2,2));
scatter(normv,normc);
axis equal;  %to illlustrate the differences in scaling, make axis 'square'

This generates the following scatter plot: 

If you change lat_d to 4, the line is less flat. 
A: I've done similar using PROC Varclus in SAS.  The basic idea is to generate a 4 cluster solution, pick the highest correlated variable with each cluster, and then to demonstrate that this 4 cluster solution explains more of the variation than the two cluster solution.  For the 2 cluster solution you could use either Varclus or the first 2 Principal Components, but I like Varclus since everything is explained via variables and not the components.  There is a varclus in R, but I'm not sure if it does the same thing.
-Ralph Winters
