I wanted to comment on Eden Trainor's post, but I need 50 reputation for that.
Eden Trainor:[...] "For example I believe that it is recommended to have mini-batch sizes related to powers of 2: "Some kinds of hardware achieve better runtime with speciﬁc sizes of arrays. Especially when using GPUs, it is common for power of 2 batch sizes to offer better runtime. Typical power of 2 batch sizes range from 32 to 256, with 16 sometimes being attempted for large models."
Thirty two as a number is your best intermediary on most computing platforms that run on 32/64/128/256 bit operation instruction sets. It won't always be like this, but functionally, in mathematics, optimization towards lowered means via axiomatic reduction is the best way to program a computer. It takes quite some skill to think like this and more importantly, to design, program and test software or micro-controller/micro-processor code like this, and as painful as it may be, it is very much worth it in the end. A good way to start learning how to think like this is to learn microcode and assembly. I never fully explored my potential in learning C and I still have to fully learn Assembly (generally). Learning Assembly will help you significantly in life, especially in getting an intrinsic understanding closer to the physical mathematics you are dealing with.
This is inherently a physics question that took me many years to really understand intrinsically, as it is a very complex problem that involves a ominlateral perspective of breaking down the problem (from the bottom up, the top down, unilaterally, etc) until you understand it inherently.
What I mean by this is that as a function of classical computing mechanics (note, this does not mean that quantum mechanics are not involved) that mini-batch size of two is not an arbitrary batch size, but a function of the precision constraint of the bimodal nature of logic gates in terms of input and output. In classical computing mechanics, you can only have with bits, the state of "ON" or "OFF", which inherently is 1 or 0, but functionally, two states which are ultimately encoded in the respective hardware.
The reason that most computers still run on 32 bit operating systems and 64 bit operating systems are inherently resource constraints that exist in society as well, but more importantly, as a technical-philosophical preference, provide the most convenience for error checking on most hardware today.
This is a function of the times we live in. Until we can develop a better case usage of optimization for larger instruction sizes (which was relatively hard for the free market to even adopt 64 bit for the longest time, but illusorily so, to a certain extent), the cost outweighs the actual benefit of developing new instruction sets, designing the mass producable hardware, etc, especially with the competition that the ARCHAIC CPU is facing in the classical-computing domain from GPUs and in general, parallelized processing unit hardware, like artificial neural networks, and other similar application-specific integrated circuitboards/processing units.
32 is also a totient function for base 10, ergo why 32 % 2, 4, 6, 8, .... 32, is always 0. This is highly convenient, if it was not obvious to you already, for computation as this conveniently maps one to one with our two natural classical computing states of "ON" and "OFF", or 0 and 1, but naturally two states respectively. This is not true for 64, which is why 32 for the time being offers the best intermediary precision.