Where does the equation $ C = 6 \times N \times T $ come from for Large Language Models, especially with a simple explanation for both passes?

Question

Why $ C = 6 \times N \times T $?

I'm trying to understand the computational steps specifically during the backward pass of neural networks in relation to the widely cited formula ( C = 6 \times N \times T ), where ( C ) is the compute cost, ( N ) is the number of parameters, and ( T ) is the number of tokens. I grasp the forward pass operations, but the backward pass, especially how gradients are computed and used for updating weights, still eludes me.

Here's the one-sentence explanation I've been given: During the backward pass, for each weight, the gradient is calculated by multiplying the gradient of the loss with respect to the output by the input value connected to that weight, and then these products are accumulated across all outputs the weight contributes to, adjusting the weight to minimize the loss.

Questions:

1. How does the backward pass operation described above lead to the multiplication by 6 in the formula ( C = 6 \times N \times T )? I'm interested in the detailed breakdown of the 2 operations in the forward pass and the 4 operations in the backward pass.

2. Could someone provide a step-by-step explanation of why multiplying ( \frac{\partial L}{\partial z_i} ) and ( x_j ) correctly computes the gradient of the loss with respect to ( W_{ij} ), and how this calculation integrates into the larger context of backpropagation?

3. In practical terms, how are these computed gradients used sequentially in a typical stochastic gradient descent algorithm to update the weights?

Insights or links to resources that provide a clear, visual, or intuitive understanding of these processes would be greatly appreciated as I aim to deepen my practical and theoretical understanding of neural network training dynamics.

Current answer a colleague gave:

Iirc for a linear layer (most of the ops) per param and token it's:

• 2 ops in the fwd pass (multiply and add)
• and 4 ops in the bwd pass (uhh multiple multiply and adds?)

ref:

• Scaling Laws for Neural Language Models Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei https://arxiv.org/abs/2001.08361
• wikipedia scaling laws https://en.wikipedia.org/wiki/Neural_scaling_law