How is the network connected in the following introductory tutorial of Pytorch?

Question

I am going over the tutorial by Pytorch

Here, they initialize a random input matrix

$$x \in \mathbb{R}^{64 \times 1000}$$ which I am assuming each row of this matrix represents a $1 \times 1000$ dimensional data and there are 64 of them.

Next, they initialize the weight between the input and the first hidden layer,

$$w_1 \in \mathbb{R}^{1000 \times 100}$$

And then they multiply these matrices as follows,

$$h = x \cdot w_1 \in \mathbb{R}^{64 \times 100}$$

Afterwards, they pass it through a Relu.

---

1. The tutorial claims that this is a fully connected network, but I just cannot see why.

Suppose in the simplest case that $x \in \mathbb{R}^{2 \times 2}$ with the components $x = \begin{bmatrix} x_1^\top \\ x_2^\top \end{bmatrix} = \begin{bmatrix} x_{11} & x_{12} \\ x_{21} & x_{22} \end{bmatrix}$ and $w_1 = \begin{bmatrix} w_1 & w_2 \\ w_3 & w_4 \end{bmatrix}$

Then multiplying $h = x \cdot w_1 = \begin{bmatrix} w_1 x_{11} + w_3 x_{12} & w_2 x_{11} + w_4 x_{12} \\ w_1 x_{21} + w_3 x_{22} & w_2 x_{21} + w_4 x_{22} \end{bmatrix} $

If I interpret each component of the matrix $h$ as an input to a Relu unit, then it is clearly not fully connected. For example, the first component (input to the first Relu unit) is $w_1 x_{11} + w_3 x_{12}$ and doesn't take into account $x_{21}, x_{22}$, which means there are edges missing.

2. What does it mean to pass a matrix $h$ to the first hidden layer? If it were a vector $\mathbb{R}^n$, then the interpretation is clear: each component of this vector corresponds to one Relu unit. Like this

But here $h$ is a matrix. What does the column, row and component/$(i,j)$th- element of this matrix represent?

Answer 1

Note that $N=64$ is the batch size, which means there are $64$ different training samples on the matrix $x$. $N$ can be any positive integer, yet the network architecture wouldn't depend on it.

In fully connected layers, each input dimension is fed into each ot the neurons in the layer. In (1), your first training sample has the dimensions $x_{11}$ and $x_{12}$, and they're multiplied with each of the two neuron's weights. So, it's fully connected. $x_{21},x_{22}$ doesn't interfere here because they belong to another training sample.

(2) Since each row of $h$ corresponds to a different training sample, passing $h$ means passing $N$ training samples of $\mathbb R^{1000}$ to the network.