GPU memory requirements of a model How do I estimate the computational(GPU-memory) requirements of a keras-tf/mxnet model theoretically before training the model on the GPU.
 A: First you count up the number of model parameters and activations. Usually the number of activations is the larger of the two (many resnet and inception-net architectures have around half as many parameters as activations, so you can try multiplying the final estimate by 1.5 as a fudge factor), and you can ignore the parameter count. 
Typically, convolution, activation, and batchnorm are all fused together in the computation graph at run time (but that depends on how well optimized the library is...), so you only store one "activation value" for this set of 3 components -- don't triple count.
Suppose your network has 100 uniform layers/feature-maps, each which is 256 by 256 and has 64 channels. This is roughly 420M activation values. (This is for a CNN, but if you are running a fully-connected or RNN model, just multiply and add up all the activation values the same way).
If you are using an optimizer like ADAM, multiply by 4 -- the network has to keep track of 4 float values for every activation: 1. the activation itself, 2. the gradient, during backprop, 3/4. the first and second moments of the gradient. If you are using SGD with momentum, multiply by 3, since second moments are not computed. Technically, it's not necessary to store the gradients of the entire network while performing backprop, but this gives us a more conservative estimate which is nice. So now we're up to 1.68 billion floats.
Now multiply by another factor of 4, since there are 4 bytes per float. This gives us 6.72 GB. Of course since model parameters take up space, and the memory allocator isn't perfect, it'll probably take up a bit more space on the GPU in practice, but this should get you within a factor of 2 of the actual memory consumption.
