How many dimensions does a neural network have?

Question

I'm asking because when we visualize something like gradient descent, we usually visualize it in two or three dimensions like so:

I understand that we do this to make visualization possible, but if we weren't thinking about visualizing it, how many dimensions would there be? Is it the number of channels at each layer in the neural network? So for ResNet50, it would be 64, 128, 256, and 512 at different points? And during the fully connected layer, it's 1000 dimensions. Are these the numbers to look at when thinking about how the optimizer works? Does the input image affect this at all?

Image credit: https://shashank-ojha.github.io/ParallelGradientDescent/

Answer 1

You're talking about the dimension of the loss function, and the answer is that the input dimension of the loss function is how many parameters and biases there are, and the output dimension is $1$.

Let's focus on that first one by looking at an example with linear regression.

$$\hat y = \hat\beta_0 + \hat\beta_1x_1+\hat\beta_2x_2+\hat\beta_3x_3 $$

Square loss is thus

$$ L\big(\hat\beta_0, \hat\beta_1, \hat\beta_2, \hat\beta_3\big) = \sum_{i=1}^n \Bigg[\bigg(y_i - \big( \hat\beta_0 + \hat\beta_1x_{i1}+\hat\beta_2x_{i2}+\hat\beta_3x_{i3} \big)\bigg)^2 \Bigg] $$

That is a function from $\mathbb{R}^4 \rightarrow \mathbb{R}$, agreed? I would call that five dimensions: four for the regression parameters and one for the output loss.

A neural network is surprisingly simular to this. The estimated value is some composition of functions, and then the loss depends on the values of the estimated parameters.

$$ \hat y =\hat b_2 + \hat w_2\text{ReLU}\big(\hat b_0 + \hat w_0x\big) + \hat w_3\text{ReLU}\big(\hat b_1 + \hat w_1x\big) $$

(I'll post that network later, but I think it would be a useful exercise to draw out what's going on. People appear quite eager to think of neural networks as being webs of circles and lines, yet there is real math going on.)

Then the loss is a function of the weights $w$ and biases $b$. Let's do absolute loss here.

$$ L\big( \hat b_0, \hat b_1, \hat b_2, \hat w_0, \hat w_1, \hat w_2, \hat w_3 \big) =$$ $$ \sum_{i=1}^n \Bigg\vert y_i - \bigg(\hat b_2 + \hat w_2\text{ReLU}\big(\hat b_0 + \hat w_0x_i\big) + \hat w_3\text{ReLU}\big(\hat b_1 + \hat w_1x_i\big)\bigg)\Bigg\vert $$

This is a function $\mathbb{R}^7 \rightarrow \mathbb{R}$. I'd call that eight dimensions.

Answer 2

Your plot shows a model with 2 parameters, because the loss $h$ is expressed as a function of $q_1, q_2$. The loss surface of a neural network is a function of each one of its parameters, so the "dimension" is the number of parameters (weights, biases) in the model.