How does the BERT model (in Tensorflow or Paddle-paddle frameworks) relate to nodes of the underlying neural-net that's being trained?

Question

The BERT model in frameworks like TensorFlow/Paddle-paddle shows various kinds of computation nodes (like subtract, accumulate, add, mult etc) in a graph like form in 12 layers.

But this graph doesn't look anything like a neural-network, one that's typically shown in textbooks (e.g. like this https://en.wikipedia.org/wiki/Artificial_neural_network#/media/File:Colored_neural_network.svg) where each edge has a weight that's being trained and there is an input layer and output layer.

Instead, when I print out the BERT graph, I can't figure out how a node in the BERT graph relates to a node in the neural-network that's being trained.

I have been using the BERT framework models to compile them to a form where we can run the model on a PC/CPU. But I still lack this basic aspect of how BERT relates to neural-net as I don't see which neural-network topology is being trained (as i'd expect topology/connections between/among various layers/nodes of the neural-net dictate how training of the neural net occurs).

Could someone explain what underlying neural-net is being trained by BERT? How do nodes in the BERT graph relate to neural-net nodes and weights on neural-net edges?

Answer 1

You appear to be referring to the view of the computation graph provided by tensorboard or a similar visualization tool.

Typically, these visualization tools don't draw every weight as a separate edge -- that would not really be feasible, since neural networks can have hundreds of millions to billions of parameters, which I doubt most plotting software could handle. So you might decide that it makes more sense to just show the "vectorized form" and have a single edge to represent a linear layer ($x \mapsto Wx$) or a bias ($x \mapsto x+b$). In this way a single neural network layer of arbitrary size could be represented with just a few edges and nodes.

However, even this is too granular to be useful -- for example modern neural networks are often composed of many "blocks", and each block may have several layers, so typically an entire block or even the entire neural network may be drawn as a single node in the graph. Since many models are built by smashing together a bunch of other neural network components as necessary, having a much higher level visualization of the computation graph is often useful.

Finally one more thing to keep in mind -- some frameworks have tools so that programmers can manually label and provide hints about how the computation graph should be organized for plotting. Without these explicit labelings, visualization tools often have to resort to heuristics and guesswork about how to draw the graph -- and the results are often quite subpar and confusing.