11
$\begingroup$

I'm asking because when we visualize something like gradient descent, we usually visualize it in two or three dimensions like so: enter image description here

I understand that we do this to make visualization possible, but if we weren't thinking about visualizing it, how many dimensions would there be? Is it the number of channels at each layer in the neural network? So for ResNet50, it would be 64, 128, 256, and 512 at different points? And during the fully connected layer, it's 1000 dimensions. Are these the numbers to look at when thinking about how the optimizer works? Does the input image affect this at all?

Image credit: https://shashank-ojha.github.io/ParallelGradientDescent/

Improve this question
$\endgroup$
3
  • $\begingroup$ How many dimensions a particular network will have depends on the network itself. It’s an easier question to ask about maximum or minimum sufficient dimensions. Sometimes there are replications and so the diagrammatic Jacobian can tell you Something about the local dimensionality. $\endgroup$ – EngrStudent Jan 20 at 0:49
  • 3
    $\begingroup$ In the context of optimizer it’s as many dimensions as the number of parameters the optimizer optimizes $\endgroup$ – Aksakal Jan 20 at 6:47
  • 1
    $\begingroup$ ResNet50 has about 25 million parameters, so for that network in particular the answer is that the loss is 25M dimensional. $\endgroup$ – Davis Yoshida Jan 20 at 18:56
13
$\begingroup$

You're talking about the dimension of the loss function, and the answer is that the input dimension of the loss function is how many parameters and biases there are, and the output dimension is $1$.

Let's focus on that first one by looking at an example with linear regression.

$$y = \beta_0 + \beta_1x_1+\beta_2x_2+\beta_3x_3 $$

Square loss is thus

$$ L\big(\beta_0, \beta_1, \beta_2, \beta_3\big) = \sum_{i=1}^n \Bigg[\bigg(y_i - \big( \beta_0 + \beta_1x_{i1}+\beta_2x_{i2}+\beta_3x_{i3} \big)\bigg)^2 \Bigg] $$

That is a function from $\mathbb{R}^4 \rightarrow \mathbb{R}$, agreed? I would call that five dimensions: four for the regression parameters and one for the output loss.

A neural network is surprisingly simular to this. The estimated value is some composition of functions, and then the loss depends on the values of the estimated parameters.

$$ y = b_2 + w_2\text{ReLU}\big(b_0 + w_0x\big) + w_3\text{ReLU}\big(b_1 + w_1x\big) $$

(I'll post that network later, but I think it would be a useful exercise to draw out what's going on. People appear quite eager to think of neural networks as being webs of circles and lines, yet there is real math going on.)

Then the loss is a function of the weights $w$ and biases $b$. Let's do absolute loss here.

$$ L\big( b_0, b_1, b_2, w_0, w_1, w_2, w_3 \big) =$$ $$ \sum_{i=1}^n \Bigg\vert y_i - \bigg(b_2 + w_2\text{ReLU}\big(b_0 + w_0x_i\big) + w_3\text{ReLU}\big(b_1 + w_1x_i\big)\bigg)\Bigg\vert $$

This is a function $\mathbb{R}^7 \rightarrow \mathbb{R}$. I'd call that eight dimensions.

Improve this answer
$\endgroup$
9
$\begingroup$

Your plot shows a model with 2 parameters, because the loss $h$ is expressed as a function of $q_1, q_2$. The loss surface of a neural network is a function of each one of its parameters, so the "dimension" is the number of parameters (weights, biases) in the model.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ So would it be correct to say that AlexNet is optimizing in a ~60M dimensional space? $\endgroup$ – jss367 Jan 20 at 0:46
  • 3
    $\begingroup$ @jss367 Yes. Likewise, Transformers are reaching into trillions territory already. $\endgroup$ – Firebug Jan 20 at 0:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.