How To Determine The Number Of Dimensions To A Machine Learning Problem

Question

I have a bit to learn about machine learning, so please pardon me if I am asking the wrong type of question. I have read some about neural networks and SVMs, so I'm not completely in the dark.

I am wondering how to tell how may dimensions a problem has: is it the number of possible outcomes or the total number of inputs? Does it depend on the type of machine learning algorithm or only on the particular problem at hand? Or am I missing something entirely?

*Most literature I have read refers to 'high-dimensionality', but I am wondering if an exact number of dimensions can be calculated. I am then hoping to use this when trying to reduce the number of dimensions (when I get that far) to judge the overall effectiveness of a strategy. But first I must better understand the dimensions of a machine learning problem.

**If necessary, please use neural networks or SVMs as a reference point, but I am also interested in hearing about genetic algorithms and anything else you might like to mention.

Answer 1

Generally the dimensionality of the problem is, as you suspected, equal to the number of inputs ( also known as, features, measurement variables ).

So in the NN model, that would be the number of nodes in the input layer. There may be unmeasured features from the problem, but normally dimensionality only refers to the measurements you have.

You may synthesise extra features from the ones you have, perhaps choosing to square one of the existing features, to make a new one, if you think this might be helpful. That would add one extra feature to the dimensionality.

On a separate point, if you are looking at SVMs you may encounter something called the Vapnik–Chervonenkis dimension ( VC dimension ). This is generalised concept which refers to the 'power', or expressiveness, of a learning algorithm. It is based on the number of points that an algorithm, with a certain set of parameters, can 'shatter" ( separate ). It is not directly related to the dimensionality of the learning problem.

Answer 2

The number of input variables is the number of dimensions.

You can reduce this number through feature extraction (i.e. creating new sets using transformations of the original set of variables, which can also augment the number of variables by the way) and/or feature selection (i.e. selecting some variables from the initial problem based on some rules, including collinearity and correlation for example).

Some algorithms are more sensitive to high dimensional data than others. SVMs are notedly robust in that aspect.

Answer 3

It is my understanding that terms number of dimensions and dimensionality refer to any system, which has some parameters. Thus, in the context of statistics, both terms, consequently, refer to any statistical model (where parameters are often referred to as variables). Therefore, determining number of dimensions of a model boils down to counting number of independent and (IMHO, separately) dependent variables, or in a preferred terminology, predictor and outcome variables. Other popular compatible terms for predictor variables include features and attributes.

As a starting point for learning more on the topic, I would suggest you reading corresponding Wikipedia articles: on dimensions, on dimensionality reduction and on the curse of dimensionality.

Finally, addressing your main concern, I would say that dimensionality of a machine learning problem is equivalent to dimensionality of a statistical model, describing said problem.