Random search for the optimal number of input features and optimal number of hidden layers for a MLP? I've performed a random search in hypothesis space $$\{(c,h)| c \in U[1,256]; h\in U[1,100];c \in \mathrm{Z} \text{ and } h \in \mathrm{Z}\}$$  that defines the parameters of a standard multilayer perceptron (MLP) neural network. 
In each step of the random search, I draw two parameters $c$ and $h$. $c$ defines the number of input features and $h$ defines the number of hidden layer nodes. $c$ and $h$ are integers drawn from a uniform distribution defined above. I train a neural network defined by $(c,h)$ and calculate a misclassification rate and average squared error rate for each model. This is done with $10$-fold cross-validation to estimate the true error for each $(c,h)$. I therefore have an average misclassification rate and an average square error rate over the train sets and the left-out sets for each parameter pair.
The question is, how do I chose the best pair of $(c,h)$ and is the method I use here sufficient? There is no reasonably clear point in the results as I'd have hoped.
The results over the hypothesis space in the training data is:

The results over the hypothesis space in the hold-out data is,

This question relates to work I've done as part of my masters dissertation, and is related to the question here this
 A: There is quite a lot of research going on right now on how to best tune the parameters of networks. One idea is to model the landscape of the generalization error with a Gaussian process and then do a good guess on what mightbe the best next set of parameters to try. Recent work includes


*

*Random Search for Hyper-Parameter Optimization, Bergstra et al,

*Algorithms for Hyper-Parameter Optimization, Bergstra et al,

*Practical Bayesian Optimization of Machine Learning Algorithms, Snoek et al.


You can find theoretical work on your approach in the first link and extensions in the other two.
An important point is that you divide your data into training, validation and test as usual. But instead of letting the human do the tuning, let it be done by the algorithm. Also, do not use the validation set in the algorithm, otherwise you might overfit.
A: I suspect that choosing amongst such a large set is likely to end up over-fitting the cross-validation error.  The CV error is a statistic estimated over a finite sample of data, and so will have a finite variance.  This means that it will be possible to make choices that reduce the CV error by making genuine improvements in generalisation, but also by exploiting the peculiarities ("nose") in the CV error due to the particular sample of data on which it is evaluated.  The more choices you make, the more likely it will be that the latter kind of reduction will take place, which will result in generalisation performance getting worse rather than better.
A better approach would be to form an ensemble from the models you have generated, perhaps weighted by their CV performance.
