I am going over the tutorial by Pytorch

enter image description here

Here, they initialize a random input matrix

$$x \in \mathbb{R}^{64 \times 1000}$$ which I am assuming each row of this matrix represents a $1 \times 1000$ dimensional data and there are 64 of them.

Next, they initialize the weight between the input and the first hidden layer,

$$w_1 \in \mathbb{R}^{1000 \times 100}$$

And then they multiply these matrices as follows,

$$h = x \cdot w_1 \in \mathbb{R}^{64 \times 100}$$

Afterwards, they pass it through a Relu.

  1. The tutorial claims that this is a fully connected network, but I just cannot see why.

Suppose in the simplest case that $x \in \mathbb{R}^{2 \times 2}$ with the components $x = \begin{bmatrix} x_1^\top \\ x_2^\top \end{bmatrix} = \begin{bmatrix} x_{11} & x_{12} \\ x_{21} & x_{22} \end{bmatrix}$ and $w_1 = \begin{bmatrix} w_1 & w_2 \\ w_3 & w_4 \end{bmatrix}$

Then multiplying $h = x \cdot w_1 = \begin{bmatrix} w_1 x_{11} + w_3 x_{12} & w_2 x_{11} + w_4 x_{12} \\ w_1 x_{21} + w_3 x_{22} & w_2 x_{21} + w_4 x_{22} \end{bmatrix} $

If I interpret each component of the matrix $h$ as an input to a Relu unit, then it is clearly not fully connected. For example, the first component (input to the first Relu unit) is $w_1 x_{11} + w_3 x_{12}$ and doesn't take into account $x_{21}, x_{22}$, which means there are edges missing.

  1. What does it mean to pass a matrix $h$ to the first hidden layer? If it were a vector $\mathbb{R}^n$, then the interpretation is clear: each component of this vector corresponds to one Relu unit. Like this enter image description here

But here $h$ is a matrix. What does the column, row and component/$(i,j)$th- element of this matrix represent?


1 Answer 1


Note that $N=64$ is the batch size, which means there are $64$ different training samples on the matrix $x$. $N$ can be any positive integer, yet the network architecture wouldn't depend on it.

In fully connected layers, each input dimension is fed into each ot the neurons in the layer. In (1), your first training sample has the dimensions $x_{11}$ and $x_{12}$, and they're multiplied with each of the two neuron's weights. So, it's fully connected. $x_{21},x_{22}$ doesn't interfere here because they belong to another training sample.

(2) Since each row of $h$ corresponds to a different training sample, passing $h$ means passing $N$ training samples of $\mathbb R^{1000}$ to the network.

  • $\begingroup$ 1. Ok I see. So fully connected doesn't mean that every component has to be connected with every (ReLU) neuron in the hidden layer, just that all the components of EACH sample are connected. $\endgroup$
    – Fraïssé
    Apr 19, 2020 at 21:11
  • $\begingroup$ 2. Suppose for the simplified example, that I only have two hidden units, one with Relu activation and one with tanh activation. Would the nonlinearity be applied to h as (a) $\begin{bmatrix} RELU(h_{11}) & Tanh(h_{12}) \\ RELU(h_{21}) & Tanh(h_{22}) \end{bmatrix}$ or (b) $\begin{bmatrix} RELU(h_{11}) & RELU(h_{12}) \\ Tanh(h_{21}) & Tanh(h_{22}) \end{bmatrix}$? $\endgroup$
    – Fraïssé
    Apr 19, 2020 at 21:14
  • $\begingroup$ In your example, $w$ is $d_1\times d_2$, where $d_1$ is layer's input dimension and $d_2$ is number of hidden units (layer's output dimension). So, each column of $w$ represents the weights of different units. Therefore, the activation function should be same columnwise, i.e. your first answer is correct. $\endgroup$
    – gunes
    Apr 19, 2020 at 21:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.