8
$\begingroup$

I have a dataset that's nominally 16-dimensional. I have about 100 samples in one case and about 20,000 in another. Based on various exploratory analyses I've conducted using PCA and heat maps, I'm convinced that the true dimensionality (i.e. the number of dimensions needed to capture most of the "signal") is around 4. I want to create a slide to that effect for a presentation. The "conventional wisdom" about this data, which I'm looking to disprove, is that the true dimensionality is one or two.

What's a good, simple visualization for showing the true dimensionality of a dataset? Preferably it should be understandable to people who have some background in statistics but are not "real" statisticians.

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Define "true", please. $\endgroup$
    – cardinal
    Mar 9, 2011 at 18:04
  • $\begingroup$ @cardinal: It's unavoidably subjective. I want the amount of dimensions that capture "most" of the variance. $\endgroup$
    – dsimcha
    Mar 9, 2011 at 18:08
  • 1
    $\begingroup$ that's fine. I just wanted to know for sure what you were going for. There are interesting examples in multivariate statistics where, e.g., if you want to find the best number of terms to keep in a least-squares sense for PCA, you should pick a smaller rank than the true rank, i.e., the rank of the actual underlying structure from which the data were generated. $\endgroup$
    – cardinal
    Mar 9, 2011 at 18:13
  • $\begingroup$ I would find a specific case where using 2-dimensions leads to a bad answer, while using 4-dimensions leads to a good answer. This will help in getting your point across - people understand examples. Add this to some of the more general stuff given below - "eblow" in the scree plot, etc. $\endgroup$
    – probabilityislogic
    Mar 10, 2011 at 23:21
  • $\begingroup$ the "examples" would be those cases which have large scores on the third and forth components $\endgroup$
    – probabilityislogic
    Mar 10, 2011 at 23:22

3 Answers 3

Reset to default
6
$\begingroup$

A standard approach would be to do PCA and then show a scree plot, which you ought to be able to get that out of any software you might choose. A little tinkering and you could make it more interpretable for your particular audience if necessary. Sometimes they can be convincing, but often they're ambiguous and there'a always room to quibble about how to read them so a scree plot may (edit: not!) be ideal. Worth a look though.

Improve this answer
$\endgroup$
9
  • 3
    $\begingroup$ @JMS, (+1) especially for the note of caution on scree plots. I would call them "standard", but also "a bad idea" (generally speaking). Here is a cautionary tale and also a way to pick the rank more safely, especially if MSE is your metric. This also provides an example, if I recall, where the true rank is the wrong one to pick for minimizing MSE. $\endgroup$
    – cardinal
    Mar 9, 2011 at 18:29
  • $\begingroup$ @cardinal (+1) Thx for the link to Owen & Perry's article. $\endgroup$
    – chl
    Mar 9, 2011 at 18:33
  • $\begingroup$ I apparently reinvented scree plots at one point but didn't realize they were called anything. Thanks for reminding me of them and letting me know that they are "standard". Since my goal is to produce a nice visualization, simple, standard and good enough wins out over complicated but more optimal. $\endgroup$
    – dsimcha
    Mar 9, 2011 at 19:08
  • 2
    $\begingroup$ @dsimcha, I did a quick Google search and there is apparently an R package that implements the approach in the link I provided. There is a nice paper from about 20 years ago (maybe 30), I believe in The American Statistician, which clearly demonstrates the flaws of inference using "standard" scree plots. I'll see if I can recall the title and will post a link or reference. While standard, but (quite) flawed, is an appealing approach, especially under time constraints, I would gently caution against such a mindset. Cheers. $\endgroup$
    – cardinal
    Mar 9, 2011 at 20:56
  • 1
    $\begingroup$ @JMS There is a paper by Zwick & Velicer (1986) that argues for using parallel analysis (resampling eigenvalues to test whether observed values were greater than expected by chance) or their MAP criterion over scree plots or the root-one rule. In their simulations the first two methods heavily outperform the others (e.g. 92% vs. 22% accuracy) in identifying the true # of components. The R package psych implements both with accompanying scree-like graphics (see fa.parallel() and VSS()). Paper is "Comparison of five rules for determining the number of components to retain." $\endgroup$
    – lockedoff
    Mar 10, 2011 at 21:52
4
$\begingroup$

One way to visualize this would be as follows:

  1. Perform a PCA on the data.
  2. Let $V$ be the vector space spanned by the first two principal component vectors, and let $V^\top$ be the complement.
  3. Decompose each vector $x_i$ in your data set as the sum of an element in $V$ plus a remainder term (which is in $V^\top$). Write this as $x_i = v_i + c_i$. (this should be easy using the results of the PCA.)
  4. Create a scatter plot of $||c_i||$ versus $||v_i||$.

If the data is truly $\le 2$ dimensional, the plot should look like a flat line.

In Matlab (ducking from all the shoes being thrown): 

lat_d = 2;   %the latent dimension of the generating process
vis_d = 16;  %manifest dimension
n = 10000;   %number of samples
x = randn(n,lat_d) * randn(lat_d,vis_d) + 0.1 * randn(n,vis_d); %add some noise
xmu = mean(x,1);
xc = bsxfun(@minus,x,xmu);    %Matlab syntax for element recycling: ugly, weird.
[U,S,V] = svd(xc);  %this will be slow;
prev = U(:,1:2) * S(1:2,1:2);
prec = U(:,3:end) * S(3:end,3:end);
normv = sqrt(sum(prev .^2,2));
normc = sqrt(sum(prec .^2,2));
scatter(normv,normc);
axis equal;  %to illlustrate the differences in scaling, make axis 'square'

This generates the following scatter plot:

scatter plot made by the code

If you change lat_d to 4, the line is less flat.

Improve this answer
$\endgroup$
10
  • $\begingroup$ @shabbychef, this seems to presuppose that (a) the true structure is a linear manifold and (b) the high variance directions are the important ones. $\endgroup$
    – cardinal
    Mar 9, 2011 at 18:09
  • 1
    $\begingroup$ @cardinal You may be confusing assumption and effect. Curvature in the manifold will be manifest as curvature in these scatterplots and the magnitude of curvature will be revealed in the magnitude of vertical variation in the plots. In effect, shabbychef is viewing the data as residing approximately within a tubular neighborhood of a low-dimensional linear space. That imposes no restrictions whatsoever on the shape of the data. $\endgroup$
    – whuber ♦
    Mar 9, 2011 at 18:57
  • $\begingroup$ @shabbyshef +1 for the decomposition idea, and of course the mentioning of the shoes :) $\endgroup$
    – mpiktas
    Mar 10, 2011 at 7:57
  • $\begingroup$ @whuber, @shabbychef, I was probably a little hasty with my initial comment, especially since I was away from a computer at the time. I think I also initially misread shabby's description and I couldn't view the plots. These make a nice way to augment a standard pairs plot. $\endgroup$
    – cardinal
    Mar 10, 2011 at 13:48
  • 1
    $\begingroup$ @cardinal Here's a 4D example generalizing your cylinder: i.imgur.com/9eF8N.png . The model is $(s,t)\to(\cos(2\pi t),\sin(2\pi t),2s/5,t^2)$ plus normal(0,.05) error added independently to all four components. The image is a scatterplot matrix of 300 iid draws with $s$ and $t$ iid uniform(0,1), ordered by decreasing variance. The proportions of total variance are 49.2%, 46.5%, 2.9%, 1.4%. The two bottom rows emulate @shabby's plots. They demonstrate the near 2D nature of the data while revealing the small nonlinearity I put into it. Is this what you were thinking of? $\endgroup$
    – whuber ♦
    Mar 10, 2011 at 21:33
0
$\begingroup$

I've done similar using PROC Varclus in SAS. The basic idea is to generate a 4 cluster solution, pick the highest correlated variable with each cluster, and then to demonstrate that this 4 cluster solution explains more of the variation than the two cluster solution. For the 2 cluster solution you could use either Varclus or the first 2 Principal Components, but I like Varclus since everything is explained via variables and not the components. There is a varclus in R, but I'm not sure if it does the same thing.

-Ralph Winters

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.