3
$\begingroup$

I've noticed that in multiple tutorials showing how to create a convolutional neural network using the MNIST dataset, the number of feature maps (or "activation maps"), chosen is typically 32. I am wondering if 32 feature maps is arbitrary or if there's an underlying purpose for using 32 specifically.

Here's an example tutorial that uses 32 feature maps in the first convolution:

https://www.tensorflow.org/get_started/mnist/pros#deep-mnist-for-experts

$\endgroup$
1
  • 1
    $\begingroup$ I doubt you'll get an answer that's any more satisfying than "32 feature maps works well for this problem." $\endgroup$
    – Sycorax
    Dec 21, 2017 at 16:54

2 Answers 2

1
$\begingroup$

This is just a hunch (I'm pretty new to ML and computer science) but I would say it is related to how the tensors are actually processed in the computer architecture (Particularly GPU architecture). For example I believe that it is recommended to have mini-batch sizes related to powers of 2: "Some kinds of hardware achieve better runtime with specific sizes of arrays. Especially when using GPUs, it is common for power of 2 batch sizes to offer better runtime. Typical power of 2 batch sizes range from 32 to 256, with 16 sometimes being attempted for large models." - Deep Learning book by Goodfellow et al.

$\endgroup$
0
$\begingroup$

I wanted to comment on Eden Trainor's post, but I need 50 reputation for that.

Blockquote Eden Trainor:[...] "For example I believe that it is recommended to have mini-batch sizes related to powers of 2: "Some kinds of hardware achieve better runtime with specific sizes of arrays. Especially when using GPUs, it is common for power of 2 batch sizes to offer better runtime. Typical power of 2 batch sizes range from 32 to 256, with 16 sometimes being attempted for large models." Blockquote

Thirty two as a number is your best intermediary on most computing platforms that run on 32/64/128/256 bit operation instruction sets. It won't always be like this, but functionally, in mathematics, optimization towards lowered means via axiomatic reduction is the best way to program a computer. It takes quite some skill to think like this and more importantly, to design, program and test software or micro-controller/micro-processor code like this, and as painful as it may be, it is very much worth it in the end. A good way to start learning how to think like this is to learn microcode and assembly. I never fully explored my potential in learning C and I still have to fully learn Assembly (generally). Learning Assembly will help you significantly in life, especially in getting an intrinsic understanding closer to the physical mathematics you are dealing with.

This is inherently a physics question that took me many years to really understand intrinsically, as it is a very complex problem that involves a ominlateral perspective of breaking down the problem (from the bottom up, the top down, unilaterally, etc) until you understand it inherently.

What I mean by this is that as a function of classical computing mechanics (note, this does not mean that quantum mechanics are not involved) that mini-batch size of two is not an arbitrary batch size, but a function of the precision constraint of the bimodal nature of logic gates in terms of input and output. In classical computing mechanics, you can only have with bits, the state of "ON" or "OFF", which inherently is 1 or 0, but functionally, two states which are ultimately encoded in the respective hardware.

The reason that most computers still run on 32 bit operating systems and 64 bit operating systems are inherently resource constraints that exist in society as well, but more importantly, as a technical-philosophical preference, provide the most convenience for error checking on most hardware today.

This is a function of the times we live in. Until we can develop a better case usage of optimization for larger instruction sizes (which was relatively hard for the free market to even adopt 64 bit for the longest time, but illusorily so, to a certain extent), the cost outweighs the actual benefit of developing new instruction sets, designing the mass producable hardware, etc, especially with the competition that the ARCHAIC CPU is facing in the classical-computing domain from GPUs and in general, parallelized processing unit hardware, like artificial neural networks, and other similar application-specific integrated circuitboards/processing units.

32 is also a totient function for base 10, ergo why 32 % 2, 4, 6, 8, .... 32, is always 0. This is highly convenient, if it was not obvious to you already, for computation as this conveniently maps one to one with our two natural classical computing states of "ON" and "OFF", or 0 and 1, but naturally two states respectively. This is not true for 64, which is why 32 for the time being offers the best intermediary precision.

Sincerely, John Lasheras

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.