3
$\begingroup$

OpenAI DALL-E is a version of GPT-3 with 12 billion parameters. Can one really estimate how many neurons are there given the number of parameters? If I understand it right, and parameters are just weights of connections between neurons, the amount or neurons should be significantly less, so it is not technically correct to compare, for example, the number of 12 billion parameters in DALL-E to the count of 86 billion neurons in human brain.

So for the sake of better analogy with humans, the question is how many neurons are there in DALL-E?

$\endgroup$

1 Answer 1

2
$\begingroup$

There's reason to believe that number of parameters, not number of neurons, is the right metric by which to measure model size. And although number of parameters isn't comparable to neurons in the human brain, it is comparable$^*$ to number of synapses in the human brain, which is estimated to be on the order of $10^{14}$.

$^*$ of course there's evidence that a single synapse can exhibit more sophisticated behavior than a single "parameter" in a neural network, so they're not completely equivalent.

$\endgroup$
3
  • $\begingroup$ +1 Would you agree that "neuron" is not a well-defined concept because, depending on how the network is wired, a neuron could repeat? I'm thinking specifically of the comparison of a rolled vs unrolled recurrent network, where the "unrolled" version repeats a neuron over and over in a time dimension. Or am I mistaken? This is a characterization I'm uncertain about. $\endgroup$
    – Sycorax
    Commented Jan 6, 2021 at 14:47
  • 1
    $\begingroup$ yeah, I guess that's another sense in which there's not a clean mapping. Although in some sense, RNN neurons are even more similar to human neurons which can spike / activate repeatedly over time. $\endgroup$
    – shimao
    Commented Jan 6, 2021 at 14:52
  • $\begingroup$ Is the structure of GPT-3 known? Maybe it is possible to get more accurate estimates of "synapse" and "neuron" count there? $\endgroup$ Commented Jan 8, 2021 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.