The impact of the number of dimensions on classification/predictive performance For the sake of simplicity, let's say we have a rectangular dataset where the columns are fields/variables and the rows are observations/events. According to the curse of dimensionality, the higher the number of fields, the poorer the classification/clustering of data points (the distance function used may not be able to cope with the high number of dimensions; all pairs of data points will be equally far/close). Of course, there are other side-effects of having high dimensions too.
However, in some algorithms and learning approaches, it seems that to improve classification/prediction is to, in fact, increase the dimensions. For example, with support vector machines we often add dimensions to make data points separable. Another example, we often try feature engineering to add dimensions to improve classification. 
So, what's the general story or reasoning behind dimensions? Does it simply depends on when to increase or reduce? Is there a general framework and/or study on determining when to add or remove features? Clearly, I have seen research to support either directions to improve classification/prediction. But I wonder if there are any more coherent studies pitting these directions against one another and understanding their behavior in general.
 A: I have dug a little deeper into this question and I think I found a satisfactory answer. The concern with increasing dimensions (or variables or features) with respect to predictive performance is tied in with the age old problem of bias-variance tradeoff. In adding more dimensions, the model will tend to increase in variance and decrease in bias, and in reducing dimensions, the model will tend to decrease in variance and increase in bias. It is not so much that adding or removing dimensions will help or hurt the predictive model's performance, but it is more of finding the right dimensions and the balance between more or less dimensions to account for the bias-variance tradeoff. 
One thing that might help others diagnose which, variance or bias, a model is suffering from, is the use of a learning curve. 
A: There is a connection between the type of classes (equivalence classes, orbits,...), more specifically their cardinality, and the dimensions(fields, columns, features,...) of the dataset. The number of distinct cardinalities of the classes is finite for a chosen dimension N. It can be found by considering the automorphism of the N-dimensional integer lattice. Results can be found in the OEIS sequence A270950. For example if N=7 then there exists exactly 29 distinct types of classes. The classification of the data is the same as the coloring of the datapoints by selecting for each datapoint one of the 29 colors. To each datapoint exist one and only one color.
