Elroch's answer is good practical advise. I thought I would mention a nice theoretical result that could be put into practice with some work.
OpenAI actually published a paper on this late last year, An Empirical Model of Large-Batch Training. They developed a statistic they call the gradient noise scale and show that it predicts the largest useful batch size.
Here's a figure from the paper.
Intuitively, for SGD-like updates, gradients of larger batches better estimate the gradient of the whole training set. However, at some point, you get diminishing returns, and computation is wasted.
Under some assumptions, they showed that the largest useful batch size (before diminishing returns) is given by
where $G$ is the matrix of gradients for all parameters and $\Sigma$ is the covariance matrix of $G$.
As explained the paper,
The noise scale is equal to the sum of the variances of the individual
gradient components, divided by the global norm of the gradient
Notice that the formula doesn't depend on the size of the training set. It does, however, depend on the gradient, which varies over training. Also, $G$ and $\Sigma$ are with respect to the whole training set and must be approximated. A method for doing this is given in the paper.