Please, how artificial neural network and support vector machine methods reduce dimensionality of variables? And based on what do we select the extraction features (retained variables)?


3 Answers 3


There is a distinction between feature selection, feature extraction, and dimensionality reduction.

Feature selection: Choosing m of n features where, m <= n.

Feature extraction: May include feature selection but can also be generative.It can create new feature dimensions based on existing features. Normally the set of generated and selected features, m, will be of a smaller size than the original feature set, n. Though potentially the new features might be used to enhance the available set of features. Normally m < n, but m >= n is possible.

Dimensionality Reduction: Any method by which a set of data with n features is represented by m features, where m < n.

Depending upon the number of output nodes chosen for the Neural Network, they are fully capable of effecting dimensionality reduction.

Consider a multi-layer neural network with N input neurons with each layer having one less neurons than the previous layer: layer sizes -> {N, N-1, N-2, ..., 1 }.

Taking the outputs of any layer offers a reduced dimensionality representation of the input space in a set of transformed features, even to the limit of representing the input space in a single dimension.

As your question was not about the primary goal of the Neural Networks but only about how they might be used for dimensionality reduction, as we have shown here it is certainly feasible that Neural Networks are capable of dimensionality reduction.

The features you get with such a reduction, like those obtained with Principal Component Analysis, are not a selection of the original features but a transformation of those features and may not have a real physical interpretation. For example consider a feature comprised of ... 0.34 Age + 0.17 Height + 0.03 Eye Color + 0.2 Make of car owned.

You may find the best performance for any given application may combine elements of feature selection and feature extraction, but this will only be revealed empirically.

  • $\begingroup$ Thanks for your help, But, it is still a little hard for me to undrestand, i am new in ANN and SVM, I have a sequence of profiles ( 250 profiles), each profile contains 1300 variables, I want to reduce the number of variables per profile I am intersted only for example by few variables, each retained variable will be plotted among 250 points=> profiles, (aim is to determine the specific profile and the specific variable which correspond to reached event) $\endgroup$
    – Sihem
    Jul 1, 2012 at 10:55
  • $\begingroup$ And i want to apply ANN and SVM to reduce the dimensionality of variables, and only to monitor few variables How i have to implement ANN and SVM and based on what criteria, i will retain the few variables $\endgroup$
    – Sihem
    Jul 1, 2012 at 10:58
  • $\begingroup$ @SihemSih Dimensionality reduction and variable selection are not exactly the same thing, as per my description. You can reduce dimensions by using techniques which transform the feature space, such as PCA or network trimming in neural networks. It is also possible to use ANNs for selection of features using the feature weights of the network. Which is your goal ... purely feature selection or dimensionality reduction? $\endgroup$ Jul 1, 2012 at 11:42
  • $\begingroup$ if i select few variables containing significant informations about the process, i think it is indirectly a dimension reduction,? so i need to make the two procedure (selection+ reduction) at the same time $\endgroup$
    – Sihem
    Jul 1, 2012 at 11:47
  • $\begingroup$ @SihemSih image_doctor is right dimensionality reduction represents an m dimensional object projected onto a lower dimensional space with the goal of maintaining most of the characetristics of the related variables. Feature selection is a method to extract a subset of features to effectly accomplish a result (such as regression or classification). Your goal with neural networks and SVM is classification. So what you are really interested in is feature selection. Your question should be what method should I use to extract the best subset of features for classification when using ANN or SVM. $\endgroup$ Jul 1, 2012 at 12:19

Who said that dimensionality reduction is the goal of these methods? Their purpose is to relate a set of feature variables to a set of classes identified in the training data so as to classify objects in their correct categories as best as possible. Dimensionality reduction would only enter into this if you are picking subsets of the features to include in the NN or the SVM. Then you could reduce the number of features used if you find that it does not have a major impact on the classification problem.

  • $\begingroup$ i want to use them, for dimension reduction, so as you told me i have to select same features? based on what criteria, i say i will retain this feature and i ignore the others? $\endgroup$
    – Sihem
    Jul 1, 2012 at 4:23
  • 1
    $\begingroup$ Why do you want to use methods intended for classification to do dimensionality reduction when there are other procedures designed for that problem (e.g. principal compnent analysis)? $\endgroup$ Jul 1, 2012 at 4:27
  • $\begingroup$ i want to compare them (used in the context of dimension reduction ) with the popular methods of dimension reduction like you mention PLS or PCA $\endgroup$
    – Sihem
    Jul 1, 2012 at 4:30
  • $\begingroup$ @SihemSih - I'm still a little unclear, do you want to achieve feature selection or feature extraction ? You mention PCA which is a feature extraction method, but you also say in another comment that you want to "select few variables..." which is feature selection. Are you clear if you want feature selection or feature extraction methods ? $\endgroup$ Jul 1, 2012 at 18:34
  • $\begingroup$ i want to reduce dimensionality of monitored variables and at the same time, the few retained variables after the reduction should be informatives about the monitored process $\endgroup$
    – Sihem
    Jul 1, 2012 at 18:56

I haven't had much interaction with ANNs in years, so I will just discuss feature selection via Support Vector Machines. The most early paper that I know that discusses feature selection via Support Vector Machines is this [1]. Basically, they achieved it by using a 1-norm regularizer instead of a 2-norm regularizer. You can probably get more details about 1-norm SVMs from [2].

It is somewhat obvious as to why 1-norm SVM should enable you to do feature selection. Since, solution to 1-norm SVM has to be sparse, in effect you end up doing feature selection.

[1] Feature Selection via Concave Minimization and Support Vector Machines

[2] 1-Norm Support Vector Machines


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.