4
$\begingroup$

I've performed a random search in hypothesis space $$\{(c,h)| c \in U[1,256]; h\in U[1,100];c \in \mathrm{Z} \text{ and } h \in \mathrm{Z}\}$$ that defines the parameters of a standard multilayer perceptron (MLP) neural network.

In each step of the random search, I draw two parameters $c$ and $h$. $c$ defines the number of input features and $h$ defines the number of hidden layer nodes. $c$ and $h$ are integers drawn from a uniform distribution defined above. I train a neural network defined by $(c,h)$ and calculate a misclassification rate and average squared error rate for each model. This is done with $10$-fold cross-validation to estimate the true error for each $(c,h)$. I therefore have an average misclassification rate and an average square error rate over the train sets and the left-out sets for each parameter pair.

The question is, how do I chose the best pair of $(c,h)$ and is the method I use here sufficient? There is no reasonably clear point in the results as I'd have hoped.

The results over the hypothesis space in the training data is:

Training sample estimated errors

The results over the hypothesis space in the hold-out data is,

Hold out sample estimated errors

This question relates to work I've done as part of my masters dissertation, and is related to the question here this

$\endgroup$
3
$\begingroup$

There is quite a lot of research going on right now on how to best tune the parameters of networks. One idea is to model the landscape of the generalization error with a Gaussian process and then do a good guess on what mightbe the best next set of parameters to try. Recent work includes

You can find theoretical work on your approach in the first link and extensions in the other two.

An important point is that you divide your data into training, validation and test as usual. But instead of letting the human do the tuning, let it be done by the algorithm. Also, do not use the validation set in the algorithm, otherwise you might overfit.

$\endgroup$
  • $\begingroup$ Optimising any criterion evaluated on a finite sample of data is likely to result in over-fitting. Even if the validation set (used for performance evaluation) is not, there can still be over-fitting, it is just that the over-fitting will not be reflected in the performance estimate (which will be biased). The random search paper is a really nice approach. (+1) $\endgroup$ – Dikran Marsupial Nov 16 '12 at 11:48
  • $\begingroup$ Thank you Bayerj and @DikranMarsupial. Bergstra et al were the research papers that guided my endeavours so far. Yet, I'm still not clear on how to select the final parameter pair to use. Because, as we can see in the second heatmap I added to the question, there is not clear approximate points where the MLP's ability start to degrade in hold out data. The estimated predictive ability in the second heat map is calculated over a hold-out sample and this results should not reflect the neural networks true generalization ability. $\endgroup$ – entropy Nov 16 '12 at 12:17
  • 1
    $\begingroup$ It looks to me that the hold out estimate is very noisy, so choosing a model is likely to lead to over-fitting and I suggested. Try looking at the area under the ROC curve, or the cross-entropy error instead, you may find that is less noisy and gives a better picture of which models are better. $\endgroup$ – Dikran Marsupial Nov 16 '12 at 12:21
  • $\begingroup$ It is not surprising that more features and more hidden units lead to lower training error. I'd suggest playing around with the visualization, e.g. trying to smooth it somehow. Training outcomes can be very noisy and have strange outliers. $\endgroup$ – bayerj Nov 16 '12 at 20:55
  • $\begingroup$ Another related idea: wouldn't it be feasible to estimate a regression $error = b_1c + b_2c^2 + b_3h + b_4h^2 + k$, and determine optimal $c$ and $h$ values over this regression function? As stated by @bayerj this might be help to smooth out the noise in hold-out data. I would however need a reference to be able to do this in my dissertation. $\endgroup$ – entropy Nov 17 '12 at 10:51
0
$\begingroup$

I suspect that choosing amongst such a large set is likely to end up over-fitting the cross-validation error. The CV error is a statistic estimated over a finite sample of data, and so will have a finite variance. This means that it will be possible to make choices that reduce the CV error by making genuine improvements in generalisation, but also by exploiting the peculiarities ("nose") in the CV error due to the particular sample of data on which it is evaluated. The more choices you make, the more likely it will be that the latter kind of reduction will take place, which will result in generalisation performance getting worse rather than better.

A better approach would be to form an ensemble from the models you have generated, perhaps weighted by their CV performance.

$\endgroup$
  • $\begingroup$ (DikranMarsupial and @bayerj) I think my out-of sample results are noisy because of my maximum number of epochs were set to low. The models therefore do not converge before stopping, especially in the case of higher $c$ and $h$ values. Do you think this could be the case? Furthermore, is CV needed for parameter selection? Can't I just evaluate my various $(c,h)$ models over a large validation set to select approximately optimal parameters. Use approximately optimal parameters and CV to report final model results. CV is slowing down the random search by a factor of ten. $\endgroup$ – entropy Nov 25 '12 at 20:31
  • $\begingroup$ Going to apply weight-decay also. But don’t want to abandon hyper-parameter search just yet. $\endgroup$ – entropy Nov 25 '12 at 20:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.