ranking neural net models with feature selection

Question

I have a sample with around 2000 observations and 10 variables which im using for classification. I plan on classifying the data with a neural net, but before doing so im using Weka's attribute selection feature to select the best combination of attributes (based on accuracy) for the model via subset evaluation.

The thing is, the attribute selection in Weka only results in the absolute best model found, and i'd like to generate a list with at least the five best ones in order of accuracy, similar to how the bestglm package in R does for generalized linear models by ranking them in order of AIC, BIC, etc.

Is there a method using R or Weka that does the same for neural nets?

Answer 1

How about trying "geneticSearch" option, in In method selection as shown.

Though I m not sure this is what you are looking for :(

Answer 2

I would advise being very careful about using feature selection with so little data and a non-linear model. There is a significant risk of over-fitting the feature selection criterion (and ending up with a model with reduced generalisation performance). Especially if you use a brittle performance metric such as classification accuracy (I'd recommend something like the Brier score instead). Make sure you keep a separate test set that you don't use in any way to fit the model (including feature selection) so you can get an unbiased measure of performance.

A better approach would be to use Bayesian Automatic Relevance Determination - see the excellent book "Neural Networks for Pattern Recognition" by Chris Bishop for details.