0
$\begingroup$

I'm using a continuous optimization algorithm for optimizing neural network's number of neurons in first and second layers besides feature selection so I used this structure for converting continues output (between 0 and 1) to positive integer values in MATLAB:

Neuron_Layer_I=(min(floor(1+21*Neuron1),21))+3; 

For example in above equation Neuron1 is output of optimization algorithm so if we have 0 for Neuron1, number of neurons in first layer is 3 and if we have 1 for Neuron1, number of neurons in first layer is 24. I have [0 24] range for second layer too. My questions :

  • Is this equation is best equation for converting my values?
  • My system tendency is toward using 24 neurons in first layer and 24 neurons for second layer. when I change (decrease or increase) ranges for number of neurons in layer 1 and layer 2 my system tendency is toward maximum values of that ranges. Why?

I have a binary classification problem and using patternnet function in MATLAB. I'm using Imperialistic Competitive Algorithm (ICA) for optimization.

Some times ago I have optimization of early stopping (max iterations) in my system besides this structure and the best early stopping max iteration was towards maximum optimization rage as we discoursed above, so I removed it. Now I have same problem with number of neurons.

Besides above questions, What is your recommendation for improving my system. My system doesn't have a specific sample-size, In my testing cases it is between 200~700 . This is a financial problem.

Thanks.

$\endgroup$
  • $\begingroup$ I distrust this. If I feed a gaussian smoothed bitmap of faces, and use your euqation to recover the 8-bit intensity values, then I will have fewer unique intensity values, but if I am working on facial recognition, then I can have substantial complexity in the integer-valued information. The number of neurons is driven by the information required to relate inputs to outputs, and not by whether the input is integer or float. $\endgroup$ – EngrStudent - Reinstate Monica Aug 18 '14 at 21:01
  • $\begingroup$ Thank you for answer. My inputs not integer values. This equation is only for converting my optimization algorithm output (continuous between 0 and 1) to an positive integer as number of neurons. This is not a transfer function for my neural network. What is your suggestion about this problem? thanks again. $\endgroup$ – user2991243 Aug 18 '14 at 21:05
1
$\begingroup$

If it is a general approach, then it applies generally. A neural network is a universal approximator. Converging on a particular number of neurons means only that it is not universal anymore.

So your equation results in the following plot. I am assuming a negative value for Neuron1 is meaningless.

enter image description here

So it is a function that maps the continuous range of 0...1 to a discrete range of 4 to 24.

"Best" does not exist without a measure of goodness. There are infinite numbers of candidate bests. For every candidate rubric there are infinite candidate anti-rubrics, and perpendicular rubriks. You have not specified a measure of goodness so there is no best to exist.

It is likely that the underlying system is better fit by a higher number of neurons. There are networks with billions of neurons. Think about your own brain, for example. Assuming it is efficient, not a bad assumption, then it should take about a billion artificial neurons to make a good approximation of what the real neurons are doing. If you tried to approximate it using 24 neurons, it would fall stunningly short - for all currently used transfer functions.

My suggestion:

  • use something like a random forest to determine something about the actual complexity of your data.
  • select neuron counts appropriate to that complexity, train properly, and cull as appropriate.

Edit/Elaboration:

A random forest subsets the data. It builds trees of random subsets of the data.

When I build a random forest, of course I use CV, but then I look at total error and total complexity as a function of tree size to determine proper tree size. I look at variable importance to reduce dimensionality, and train using the important variables. Once you have a reasonable tree, you can iterate on training-validation using progressively smaller chunks of the data for training until you hit a "cliff" or "knee" in representation. At that point you are at minimum rows and minimum columns. You can double-check leaf size with an eye for minimization, but you are reasonably well situated to apply the fitting with a NN. You want to do a few reruns with random row-selection at every level of split, of course, to make sure that your mean error at the split-size is not an artifact.

This leaves you several estimates of parameter:

$ #rows x #colums = minimum data element count$ $ #leaves x #trees = tree parameter count $

Then you can look at the parameters count for your neurons and put a ceiling there. If you can't beat the trees with the NN you have a problem. You add up all the "weights" and "biases" in your ANN and that is your parameter count. You can also look at this to inform your training set, and to subsample your original data if training would take a prohibitively long time.

So you start your ANN with the same number of parameters as the RF. You train until suitable representation occurs. Then you perform a cull operation, where you cull the least informative neuron, and update the training. You track the fully-trained error as a function of parameter count and make a good judgment call as to where you are comfortable leaving your NN. I would suggest evaluating performance from RF-base to at least 25% fewer parameters than RF-base as a domain to evaluate fully-trained performance.

In my personal experience, many things can be approximated with Random Forests of less than 300 trees, otherwise built to the recommendations of Breiman and company. Sometimes fewer than 50 trees will do the job well. This means some ballpark complexity ceilings are between 300-1800 parameters. Your actual mileage is going to depend entirely on your data.

aside:

This would make a great (winning) science fair (ISEF) project. As a former judge - I would have liked to see some of this there, in the applied mathematics area. If someone wanted to apply this to a dozen or so textbook data sets and compare to textbook results. Given that "machine learning" and "data-mining" are pretty popular right now this could be good.

$\endgroup$
  • $\begingroup$ Thank you for your answer. I'm using classification accuracy as cost function of my optimization algorithm. My model is searching to find best structure of neural network (inputs and number of neurons) based on classification accuracy + I'm using 5-fold cross validation and this accuracy is on the test samples. With these information have i same problem? Can you describe more your suggestions? Thank you so much for your helps. I think random forest is a structure for decision trees and I'm not familiar with second suggestion. $\endgroup$ – user2991243 Aug 19 '14 at 6:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.