I am studying neural network. I read a lot about mini-batch gradient descent. So now, I was implementing it in TensorFlow.

But I have some questions regarding the matrix formulation of the problem/vectorization.


First some notation $w_{ij}^l$ is the weight on the connection connecting node $j$ in layer $l$ to node $i$ in layer $l+1$.

If we have $4$ nodes and $3$ features we could have:

$W^{(1)} = \begin{bmatrix} w_{11}^{(1)} & w_{12}^{(1)} & w_{13}^{(1)} \\ w_{21}^{(1)} & w_{22}^{(1)} & w_{23}^{(1)} \\ w_{31}^{(1)} & w_{32}^{(1)} & w_{33}^{(1)} \\ w_{41}^{(1)} & w_{42}^{(1)} & w_{43}^{(1)} \end{bmatrix}$

At the same time, suppose we have a batch size of $5$, then the input matrix might be

$X = \begin{bmatrix} x_{11} & x_{12} & x_{13} \\ x_{21} & x_{22} & x_{23} \\ x_{31} & x_{32} & x_{33} \\ x_{41} & x_{42} & x_{43} \\ x_{51} & x_{52} & x_{53} \end{bmatrix}$

where $x_{ij}$ is the input in sample $i$ of feature $j$.

What I don't understand

Everywhere I go I see $f(W^TX +\mathbb{b})$ as the formula. However this clearly doesn't make sense here, as we would have incompatible shapes!! $W$ has shapes $(\text{number nodes}, \text{number features})$, while $X$ has shape $($ batch size, number features $)$.

Ideally we would like, for each sample, to have something like this (I only write the first two columns). But how do we get there? Is this the correct way?

$A^{(1)} = \begin{bmatrix} w_{11}^{(1)}x_{11} + w_{12}^{(1)}x_{12} + w_{13}^{(1)}x_{13} & w_{11}^{(1)}x_{21} + w_{12}^{(1)}x_{22} + w_{13}^{(1)}x_{23}\\ w_{21}^{(1)} x_{11} + w_{22}^{(1)}x_{12} + w_{23}^{(1)}x_{13} & w_{11}^{(1)}x_{21} + w_{12}^{(1)}x_{22} + w_{13}^{(1)}x_{23}\\ w_{31}^{(1)}x_{11} + w_{32}^{(1)}x_{12} + w_{33}^{(1)}x_{13} & w_{31}^{(1)}x_{31} + w_{32}^{(1)}x_{32} + w_{33}^{(1)}x_{33}\\ w_{41}^{(1)}x_{11} + w_{42}^{(1)}x_{12} + w_{43}^{(1)}x_{13} & w_{41}^{(1)}x_{41} + w_{42}^{(1)}x_{42} + w_{43}^{(1)}x_{43} \end{bmatrix}$


1 Answer 1


I don't see the problem. Here's a neural net with three layers: $$ \begin{array}{c} y = \beta_0 + \mathbf{V}_1\beta \\ \mathbf{V}_1 = a(\gamma_1 + \mathbf{V}_2\Gamma_1)\\ \mathbf{V}_2 = a(\gamma_2 + \mathbf{V}_3\Gamma_2)\\ \mathbf{V}_3 = a(\gamma_3 + \mathbf{X}\Gamma_3) \end{array} $$

Where $\mathbf{X}$ is your data.

Say that $\mathbf{X}$ is $N \times p$. That means that $\Gamma_3$ is $p \times p_3$, $\mathbf{V}_3$ is therefore $N \times p_3$ and $\Gamma_2$ is $p_3 \times p_2$, $\mathbf{V}_2$ is therefore $N \times p_2$ and $\Gamma_1$ is $p_2 \times p_1$, and $\beta$ is $p_1 \times 1$ (assuming a univariate response).

For a minibatch, just swap out $N$ with $B<N$. It doesn't change the dimensions of the weights, just that of the derived variables/hidden layers $\mathbf{V}$

Your problem is that you were thinking of $X$ as row-oriented vs column oriented, I think. That's why your matrices weren't conformable. But minibatch has nothing to do with that -- minibatch chops off observations, not features. Dropout, on the other hand, chops off features. In so doing it propagates the deleted columns upward/downward on the forward backward passes.

  • 3
    $\begingroup$ Inevitably! I am of the opinion that the tensor-based and index based way that neural nets are taught makes them seem needlessly complex, especially to people with a wit of math background. $\endgroup$ Oct 20, 2017 at 14:14
  • 1
    $\begingroup$ By knowing $X$ is $N \times p$ and $\Gamma_3$ is $p\times p_3$, we know that $V_3$ is $N \times p_3$ $\endgroup$ Oct 20, 2017 at 14:22
  • 1
    $\begingroup$ IDK but I'm fairly confident that you could use commutative rules for matrices to make this work in row-wise representations of data. $\endgroup$ Oct 20, 2017 at 14:25
  • 1
    $\begingroup$ Yeah. Editing.. $\endgroup$ Oct 20, 2017 at 14:41
  • 1
    $\begingroup$ Any tensor can be represented in two dimensions. If you are writing code in assembly or for a GPU, the tensor representation might be optimal. But if you're relying on either blas or sparse matrix libraries, I think the 2d representation is better. But ultimately the two are mathematically equivalent. $\endgroup$ Oct 20, 2017 at 16:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.