3
$\begingroup$

I always found the concept of determining the "ideal" number of components/factors for an ICA/PCA/FA via a scree plot useful and quick, but also a bit shaky.

In an effort to try to make the scree plot "elbow" more obvious (and without trying to fit a function to the scree plot data points). I was thinking whether plotting on log-scale coordinates might help shed some light.

enter image description here

The non-log plot (first from the left) is pretty bunched up, but it's as "obvious" as it gets with scree plots that the third factor is the elbow (and thus my ideal number of factors should be 3).

Now, it also seems that whether my log scale is in base 2 or 10 makes little difference for the plot appearance - which I guess is good.

Lastly, I find that the X/Y log plots show the most quaintness for the 3rd factor, making it the point where 2 types of "curves" in the plot meet. I also think this feature stands out better here than in the first (linear) plot.

Is it safe to assume this is always the case? Is it worth recommending to use X/Y log coordinates for scree plots?

Improve this question
$\endgroup$
12
  • 1
    $\begingroup$ For your information: there exist two expositions of the Cattell scree-plot rule: If the "elbow" is the m-th eigenvalue, (1) choose to extract m components; or (2) choose to extract m-1 components. Nobody knows which's better - because the whole rule is so much heuristic. $\endgroup$
    – ttnphns
    Aug 8, 2014 at 2:17
  • 1
    $\begingroup$ The core idea is a logical one: noise components, because they are noise, should have variances decreasing approximately linearly (while signal components may decrease their variances at any rate). To detect it, one shouldn't transform the plot. At least, I don't see any particular good of it. But - as you wish... $\endgroup$
    – ttnphns
    Aug 8, 2014 at 2:25
  • 2
    $\begingroup$ @ttnphns: Why do you think that the variance of noise components should decrease linearly? In fact, I analyzed it once and it's more of a power-law decay in the middle (and quite messy overall). $\endgroup$
    – amoeba
    Aug 8, 2014 at 7:37
  • 1
    $\begingroup$ @amoeba, you may be right. I missed that your answer somehow. I'll check it myself, simulating. Anyway, I believe Cattell didn't do statistical probes for that. Note also that often people don't plot scree for all the many eigenvalues, this way they might observe only the linear part of the noise curve. $\endgroup$
    – ttnphns
    Aug 8, 2014 at 8:05
  • 1
    $\begingroup$ This is questionnaire design data with as many questions as you see eigenvalues. So, @amoeba, what do you think about my log plotting? is there any other way to improve scree plot "obviousness" (like with outlier testing) or should I forget about it and use some other method (like Horn's Parallel Analysis)? $\endgroup$
    – TheChymera
    Aug 8, 2014 at 17:40

1 Answer 1

Reset to default
1
$\begingroup$

Summary: I think that converting y-axis to log scale often works the best, in your example as well.

Continuing what has already been said in the comments above: The main idea of looking at the spectrum (also known as scree plot, but I am not used to factor analysis and am not a big fan of its terminology) to identify the number of significant components, is to see how many eigenvalues "stand out" of the "bulk". Therefore, for example, looking at your first plot, I would definitely conclude that the answer is two, and not three.

Does it make sense to transform the spectrum in any way? I would say yes, but only if it helps achieving the above goal, i.e. makes it more obvious which eigenvalues stand out. The first (untransformed) plot is suboptimal, because one simply cannot see anything starting with the third eigenvalue, so there is definitely room for improvement.

Looking at your plots, I would say that transforming eigenvalues (y-axis) to log scale works the best, and in particular better than the log-log plot, mainly because the "bulk" of the spectrum becomes nicely linear, and makes it easy to see how many eigenvalues are strongly above this linear trend (answer: two definitely, but maybe three).

Interestingly, linear trend on a log-plot corresponds to exponential decay, and this is something I have observed myself in many different datasets. The reason for this is not entirely clear to me; please check my answer in Eigenvalues of correlation matrices exhibit exponential decay -- there is an expression for the spectrum of a random covariance matrix, but it is not given by an exponential and I am not exactly sure why exponential function gives such a good approximation. Still, it seems to be the case.

Improve this answer
$\endgroup$
6
  • $\begingroup$ many thanks for the answer, however, I cannot agree that the y-log plot makes anything more obvious. You say it shows a log-linear decay for the noise components - which is true for the middle of the noise spectrum. Around the elusive elbow, though, it shows some weird curving up making it non-obvious for me where in the (1,15] interval the elbow is supposed to be... $\endgroup$
    – TheChymera
    Aug 8, 2014 at 22:30
  • $\begingroup$ @TheChymera: Well, it is obviously a matter of judgment, and I have provided mine. If you want a more "objective" criterion, do Monte Carlo permutation test (aka parallel analysis), or cross-validation, or anything like that. Having said that, 1 to 15, really? Come on, it is either 2 or 3, the rest just blends together completely. $\endgroup$
    – amoeba
    Aug 8, 2014 at 23:06
  • $\begingroup$ that's true (that everything else blends together) - but am I not supposed to be looking for an elbow? Or should I only be looking at the distanced between the points in the log plot? $\endgroup$
    – TheChymera
    Aug 8, 2014 at 23:25
  • $\begingroup$ @TheChymera: But what is an elbow? For me, the telling sign of noise components is precisely that they blend together; the shape of the noise spectrum is not very clear (Cattell, who coined the term "scree plot") wrote about linear decay, hence "scree"; to me it looks rather exponential; but as you point out, not exactly so; so it's complicated and might depend on the data as well), so defining the elbow via the shape of the spectrum does not seem right. I suggest looking for components that stand out from the rest. $\endgroup$
    – amoeba
    Aug 9, 2014 at 8:54
  • 1
    $\begingroup$ @TheChymera: To tell the truth, this idea never crossed my mind. I agree that it could be a curious investigation (if it does both: analyzes a lot of real data sets from different fields, and presents some extensive modeling), but I don't think I am going to have time for this... I might think about it though. If you get any further ideas about your spectra, let me know, I am curious what you will settle on. Hope our exchange was useful for you. $\endgroup$
    – amoeba
    Aug 10, 2014 at 21:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.