# Why do real-world high dimensional data often have much lower inherent dimensionality?

In a previous year exam about data analysis, there was a question that I'm confused about.

Here's the question:

Explain why it is natural to expect that in real-world high dimensional data the inherent dimensionality of data points will be smaller than the number of coordinates. How is the smaller number of relevant new co-ordinate axis found in PCA?

I'm confused about the real-world high dimensional data being smaller than the number of coordinates. I'd really appreciate if someone can point me in the right direction.

It's not that the dimensionality of the data (more precisely, the statistical process at work) is smaller than the coordinate space, it's that there is often not enough available data to get statistically significant results in every direction. This is a manifestation of the famed curse of dimensionality. PCA is an attempt to determine the directions in which you have the best shot at getting reliable and stable results - it does this by finding the directions in which the data is most spread out.

Of course there are random processes that do only occur in a low dimensional subspace of the feature space, but in general it's best not to assume this is so without evidence. Nonetheless, in any situation, you are bound to do the best you can with the data you have, and the data you have can only support a finite number of inferences.

The answer is relatively simple. Typically for a real world problem you don't know which features to use for the problem. Very often you end up throwing too many features and let the algorithm figure out which ones are discriminative.

Let's take MNIST digit classification as an example. You are given 28x28 black&white images centered on the digit. You have a choice: build the classifier based on the individual pixel values (easy but meaning more dimensions) or come up with more intelligent features (harder but less dimensions). If you go with individual pixel values, you know that you don't need those 784 dimensions to encode differences between 10 digits. This information is buried somewhere inside.

• I don't see how this answers the question. Perhaps you don't need those 784 dimensions to discriminate between 10 digits, but how does it imply that all MNIST images are confined to a submanifold with $d\ll 784$ dimensions? This is what OP is asking about. – amoeba May 5 '15 at 22:09
• @amoeba if we knew a priori which features are the good ones for the task, we should not be playing with dimensionality reduction tasks. At least in many real world applications. Perhaps I still don't understand the question. – Vladislavs Dovgalecs May 5 '15 at 22:12
• As I understand the question, it has nothing at all to do with classification. Take the MNIST images, it's a good example. Each image is a point in 784-dimensional space. Let's take one billion of images, they make up a cloud of points. Here is the question: is it true that all these points are actually confined to a subspace of much lower dimensionality; if so, why? It can be natural images instead of digits, so number of classes (or even existence of classes) does not matter here. – amoeba May 5 '15 at 22:16
• I don't agree with that. Data points form a set $S \subset \mathbb R^n$ in some large $n$-dimensional Euclidean space. Intrinsic dimensionality is a certain geometrical property of this set (even though I will agree that this property is only vaguely defined); "prior knowledge" is irrelevant here. We don't need to know that the data are actually from MNIST in order to estimate the intrinsic dimensionality. – amoeba May 6 '15 at 20:43
• @amoeba More reading is always good. From literature I see that this is an intrinsic property, not necessarily linked to some prior knowledge. It looks like in pattern recognition domain (for image classification) the term definition is different or misused. I acknowledge my mistake. – Vladislavs Dovgalecs May 6 '15 at 21:29

What "the inherent dimensionality of data points will be smaller than the number of coordinates" means is this: If you have 2 dimensional data, you need a 2 dimensional coordinate system (x,y for example) in order to be able to show them. if you data has n dimensions, you need an n dimensional coordinate system, where after n=3 it is impossible for us to visualize that geometrically (at least for me). So if you have high dimensional data, it is diffucult to show it. However, there is a trick to that. What you can do is to say: OK, I might need a lot of dimensions to show the data in the usual corrdinate system, but is there a way to transform the data into some other form, so I will need less axis to show it, or similarly an alternative coordinate system, where I can show my data with less axis? The answer is yes, and PCA is how you do it. You say, OK, let me find the direction where the change in my data is the highest, i.e. the varience is highest, and use this direction as the first axis of my new coordinate system. Theen you say, OK, now I have a direction that explains some (most) of the variance in my data, but what in which direction is the variance of the data second highest, you find it and it is the second direction of your coordinate system. Then you repeat it several times and come up with several axes that explain the most of the variance of your data. The rest of the varience in your data is very small and not so relevant, as you already have most of the change in your data. So now, in this new system, you have nearly all the variance of your original data, however you have less axes, i.e. your data is nearly as well represented in this new corrdinate system as in the previous, however it has significantly less axes in the new one. This is what is meant by your data having inherently less dimensions (in the new coord. system) than the number of coordinates (in the old coord. system).

This representation in the lower dimensional new coordinate system is allowed by the fact that your data actually has some dependencies in the previous corrd. systems bases, i.e. your data can be more efficiently represented in another coord. system with other basis functions. PCA is a special way of finding these basis functions that define the new coordinate system.

In real world everything's controlled by God's. That's why everything's dependent on God. So, everything's really in one dimension, the dimension of God's will. This is is the only true answer, but I doubt that your Prof will accept it. So, here's an easier one.

In real world we're probably not gathering some random data. We usually collect data trying to solve some problem. When we do this it's most likely that we'll be looking for the data that is related to the phenomenon of interest. By virtue of this relation it's likely that all the different data points are measuring the same thing from different angles, figuratively. And that thing is probably what interests us in the first place. So, if we somehow can extract the essence of the phenomenon, we'll probably end up with a fewer number of dimensions that all these variables.

The information in the high-dimensional space can often be captured by a smaller number of latent variables/dimensions because there tends to be dependence (e.g. multicollinearity) between variables in the high-dimensional space.

There tends to be dependence because many of the variables are probably going to form networks of causal connections. I am thinking of biological contexts such as genomic or neuroimaging data, for example. Maybe the situation is different in other domains such as astronomy, however.

The statement in the exam question seems contentious. Take a simple example, like plotting life expectancy of a baby versus various factors including average daily salt consumption. The baby will (on average) die early if the salt consumption is near zero, will thrive it the salt consumption is intermediate, and will die immediately if it's very large. PCA is likely to obscure this crucial information.

Usually, if a phenomenon cannot be explained using only a few factors, then it will be inaccessible to the human mind, and too complex to publish. Then we wait until someone in the future thinks of a way to explain the phenomenon simply, perhaps using completely different variables.

With real data, it's certainly true that one usually finds that PCA gives rise to a coordinate change, indicating certain directions as being more significant. If these indications cannot be backed up with specifically designed experiments or with existing theory or with previous results, then the research may be discontinued. The overall result may be a bias in the assumptions made by examiners as to the nature of the scientific enterprise.