Factor analysis and curved manifolds I am self-studying Kevin Murphy's book, and one passage on factor analysis states that "the Factor Analysis model assumes that the data lives on a low dimensional linear manifold. In reality - most data is better modelled by some form of low dimensional curved manifold". Can anyone share some intuition what this means, and why this would be the case? Many thanks in advance.
 A: A manifold is an $m$-dimensional surface that lives in an $n$-dimensional space, where $m < n$.
Here's a simple example. Suppose you only have two variables, height and weight. They are of course correlated, with taller people also being heavier on average. If you plotted these variables for a large number of people, you might have a chart that looks like this:

Notice how, to a (very) rough approximation, the points fall on a straight line. There is some scatter around that line, but the relationship is clear. A straight line is an example of a one-dimensional linear manifold. In this case, the manifold is embedded inside a two-dimensional space since we're dealing with two variables, but in general, the dimensionality of the embedding space can be much greater than that of the manifold itself. If we had a third variable that also had a strong linear correlation with height and weight, say waist girth, then the scatter cloud of all 3 variables would be a one-dimensional manifold in 3-dimensional space.
Now suppose instead of height and weight, we have height and age. If we did a similar plot of these variables, we might get something rather different:

In this case, the plot levels off above a certain age, because people don't keep growing indefinitely. This is a curved manifold, because it can't be described as a simple linear relationship betweeen the two variables.
What the book is saying is that in practice, data is more likely to be of the second form rather than the first. Now that said, people often make a working assumption that a linear manifold is the underlying truth, because it makes the modelling and interpretation a lot simpler. With height and weight, you can say that increasing height leads to increasing weight (or vice-versa). In general, such one-line descriptions aren't possible with nonlinear relationships.
Depending on the data and the aims of the analysis, assuming all your variables are linearly related may not be that bad. Even in the second example above, notice how, on the left side of the chart, height and age are approximately linearly related. There is a similar linear relationship on the right side, but with a different slope. So, if you were willing to focus only on one part of the overall graph, you might not go wrong in making a linearity assumption. But in this case, you should still be aware that you're only looking at a part of the overall picture, and that extrapolating from your data could lead to incorrect conclusions.
