Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite
1

Please, how artificial neural network and support vector machine methods reduce dimensionality of variables? And based on what do we select the extraction features (retained variables)?

share|improve this question

3 Answers

up vote 1 down vote

I haven't had much interaction with ANNs in years, so I will just discuss feature selection via Support Vector Machines. The most early paper that I know that discusses feature selection via Support Vector Machines is this [1]. Basically, they achieved it by using a 1-norm regularizer instead of a 2-norm regularizer. You can probably get more details about 1-norm SVMs from [2].

It is somewhat obvious as to why 1-norm SVM should enable you to do feature selection. Since, solution to 1-norm SVM has to be sparse, in effect you end up doing feature selection.

[1] Feature Selection via Concave Minimization and Support Vector Machines

[2] 1-Norm Support Vector Machines

share|improve this answer
up vote 5 down vote

There is a distinction between feature selection, feature extraction, and dimensionality reduction.

Feature selection: Choosing m of n features where, m <= n.

Feature extraction: May include feature selection but can also be generative.It can create new feature dimensions based on existing features. Normally the set of generated and selected features, m, will be of a smaller size than the original feature set, n. Though potentially the new features might be used to enhance the available set of features. Normally m < n, but m >= n is possible.

Dimensionality Reduction: Any method by which a set of data with n features is represented by m features, where m < n.

Depending upon the number of output nodes chosen for the Neural Network, they are fully capable of effecting dimensionality reduction.

Consider a multi-layer neural network with N input neurons with each layer having one less neurons than the previous layer: layer sizes -> {N, N-1, N-2, ..., 1 }.

Taking the outputs of any layer offers a reduced dimensionality representation of the input space in a set of transformed features, even to the limit of representing the input space in a single dimension.

As your question was not about the primary goal of the Neural Networks but only about how they might be used for dimensionality reduction, as we have shown here it is certainly feasible that Neural Networks are capable of dimensionality reduction.

The features you get with such a reduction, like those obtained with Principal Component Analysis, are not a selection of the original features but a transformation of those features and may not have a real physical interpretation. For example consider a feature comprised of ... 0.34 Age + 0.17 Height + 0.03 Eye Color + 0.2 Make of car owned.

You may find the best performance for any given application may combine elements of feature selection and feature extraction, but this will only be revealed empirically.

share|improve this answer
Thanks for your help, But, it is still a little hard for me to undrestand, i am new in ANN and SVM, I have a sequence of profiles ( 250 profiles), each profile contains 1300 variables, I want to reduce the number of variables per profile I am intersted only for example by few variables, each retained variable will be plotted among 250 points=> profiles, (aim is to determine the specific profile and the specific variable which correspond to reached event) – Sihem Jul 1 '12 at 10:55
And i want to apply ANN and SVM to reduce the dimensionality of variables, and only to monitor few variables How i have to implement ANN and SVM and based on what criteria, i will retain the few variables – Sihem Jul 1 '12 at 10:58
@SihemSih Dimensionality reduction and variable selection are not exactly the same thing, as per my description. You can reduce dimensions by using techniques which transform the feature space, such as PCA or network trimming in neural networks. It is also possible to use ANNs for selection of features using the feature weights of the network. Which is your goal ... purely feature selection or dimensionality reduction? – image_doctor Jul 1 '12 at 11:42
if i select few variables containing significant informations about the process, i think it is indirectly a dimension reduction,? so i need to make the two procedure (selection+ reduction) at the same time – Sihem Jul 1 '12 at 11:47
@SihemSih image_doctor is right dimensionality reduction represents an m dimensional object projected onto a lower dimensional space with the goal of maintaining most of the characetristics of the related variables. Feature selection is a method to extract a subset of features to effectly accomplish a result (such as regression or classification). Your goal with neural networks and SVM is classification. So what you are really interested in is feature selection. Your question should be what method should I use to extract the best subset of features for classification when using ANN or SVM. – Michael Chernick Jul 1 '12 at 12:19
show 2 more comments
up vote 1 down vote

Who said that dimensionality reduction is the goal of these methods? Their purpose is to relate a set of feature variables to a set of classes identified in the training data so as to classify objects in their correct categories as best as possible. Dimensionality reduction would only enter into this if you are picking subsets of the features to include in the NN or the SVM. Then you could reduce the number of features used if you find that it does not have a major impact on the classification problem.

share|improve this answer
i want to use them, for dimension reduction, so as you told me i have to select same features? based on what criteria, i say i will retain this feature and i ignore the others? – Sihem Jul 1 '12 at 4:23
1  
Why do you want to use methods intended for classification to do dimensionality reduction when there are other procedures designed for that problem (e.g. principal compnent analysis)? – Michael Chernick Jul 1 '12 at 4:27
i want to compare them (used in the context of dimension reduction ) with the popular methods of dimension reduction like you mention PLS or PCA – Sihem Jul 1 '12 at 4:30
@SihemSih - I'm still a little unclear, do you want to achieve feature selection or feature extraction ? You mention PCA which is a feature extraction method, but you also say in another comment that you want to "select few variables..." which is feature selection. Are you clear if you want feature selection or feature extraction methods ? – image_doctor Jul 1 '12 at 18:34
i want to reduce dimensionality of monitored variables and at the same time, the few retained variables after the reduction should be informatives about the monitored process – Sihem Jul 1 '12 at 18:56
show 4 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.