2
$\begingroup$

I have a bit to learn about machine learning, so please pardon me if I am asking the wrong type of question. I have read some about neural networks and SVMs, so I'm not completely in the dark.

I am wondering how to tell how may dimensions a problem has: is it the number of possible outcomes or the total number of inputs? Does it depend on the type of machine learning algorithm or only on the particular problem at hand? Or am I missing something entirely?

*Most literature I have read refers to 'high-dimensionality', but I am wondering if an exact number of dimensions can be calculated. I am then hoping to use this when trying to reduce the number of dimensions (when I get that far) to judge the overall effectiveness of a strategy. But first I must better understand the dimensions of a machine learning problem.

**If necessary, please use neural networks or SVMs as a reference point, but I am also interested in hearing about genetic algorithms and anything else you might like to mention.

$\endgroup$
0
$\begingroup$

Generally the dimensionality of the problem is, as you suspected, equal to the number of inputs ( also known as, features, measurement variables ).

So in the NN model, that would be the number of nodes in the input layer. There may be unmeasured features from the problem, but normally dimensionality only refers to the measurements you have.

You may synthesise extra features from the ones you have, perhaps choosing to square one of the existing features, to make a new one, if you think this might be helpful. That would add one extra feature to the dimensionality.

On a separate point, if you are looking at SVMs you may encounter something called the Vapnik–Chervonenkis dimension ( VC dimension ). This is generalised concept which refers to the 'power', or expressiveness, of a learning algorithm. It is based on the number of points that an algorithm, with a certain set of parameters, can 'shatter" ( separate ). It is not directly related to the dimensionality of the learning problem.

$\endgroup$
1
$\begingroup$

It is my understanding that terms number of dimensions and dimensionality refer to any system, which has some parameters. Thus, in the context of statistics, both terms, consequently, refer to any statistical model (where parameters are often referred to as variables). Therefore, determining number of dimensions of a model boils down to counting number of independent and (IMHO, separately) dependent variables, or in a preferred terminology, predictor and outcome variables. Other popular compatible terms for predictor variables include features and attributes.

As a starting point for learning more on the topic, I would suggest you reading corresponding Wikipedia articles: on dimensions, on dimensionality reduction and on the curse of dimensionality.

Finally, addressing your main concern, I would say that dimensionality of a machine learning problem is equivalent to dimensionality of a statistical model, describing said problem.

$\endgroup$
  • $\begingroup$ So you are saying the nuimber of dimensions is increased from the standard independent and dependent variables due to the whole 'probability factor'? Is there a new dimension for every possible value of the variable (or even worse, for every combination of all the variables) within the defined possible range of probable values? For example, if there is a probability that a variable x will be between 5 and 8, and variable y will be between 15 and 17... then we have 3 dimensions for the range of x, and 2 dimensions for the range of y, so the problem will have a total of 5 dimensions? $\endgroup$ – user58446 Apr 10 '15 at 19:29
  • $\begingroup$ @user58446: I am not sure what do you mean by "whole probability factor". But, yes, I think that the total dimensionality of a problem/model/system is a sum of individual dimensions across all variables. $\endgroup$ – Aleksandr Blekh Apr 10 '15 at 22:36
  • $\begingroup$ The 'whole probability factor' refers to the schotastic, as opposed to deterministic, nature of the problem. However, my example in the comment may be further altered by suggesting that the range for x may be between 5 and 8, but the probability for x having specific individual values is as follows: 5 = 20%, 6 = 40%, 7 = 10%, 8 = 30%. Likewise, for y: 15 = 5%, 16 = 80%, 17 = 15%. How do these probabilities affect the dimensionality of the problem, if at all? $\endgroup$ – user58446 Apr 10 '15 at 22:47
  • $\begingroup$ @user58446: I still don't see how those probabilities affect the dimensionality of the problem. If you think otherwise, please refer to definition of dimensionality, which includes the probability aspect. $\endgroup$ – Aleksandr Blekh Apr 10 '15 at 23:00
  • $\begingroup$ You misunderstand: I was taking a guess at what you meant by your explanation. You mentioned that "dimensionality of a machine learning problem is equivalent to dimensionality of a statistical model", or schotastic model, which involves probability unlike a deterministic system. You also said that the dimensionality of the problem "is a sum of individual dimensions across all variables". So, I comined the range of the values (from second comment) and combined it with the stochastic model which involves probabilities for different values within the aforementioned range of possible values... $\endgroup$ – user58446 Apr 10 '15 at 23:08
1
$\begingroup$

The number of input variables is the number of dimensions.

You can reduce this number through feature extraction (i.e. creating new sets using transformations of the original set of variables, which can also augment the number of variables by the way) and/or feature selection (i.e. selecting some variables from the initial problem based on some rules, including collinearity and correlation for example).

Some algorithms are more sensitive to high dimensional data than others. SVMs are notedly robust in that aspect.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.