# mini batch matrix form of neural network

I am studying neural network. I read a lot about mini-batch gradient descent. So now, I was implementing it in TensorFlow.

But I have some questions regarding the matrix formulation of the problem/vectorization.

Notation

First some notation $w_{ij}^l$ is the weight on the connection connecting node $j$ in layer $l$ to node $i$ in layer $l+1$.

If we have $4$ nodes and $3$ features we could have:

$W^{(1)} = \begin{bmatrix} w_{11}^{(1)} & w_{12}^{(1)} & w_{13}^{(1)} \\ w_{21}^{(1)} & w_{22}^{(1)} & w_{23}^{(1)} \\ w_{31}^{(1)} & w_{32}^{(1)} & w_{33}^{(1)} \\ w_{41}^{(1)} & w_{42}^{(1)} & w_{43}^{(1)} \end{bmatrix}$

At the same time, suppose we have a batch size of $5$, then the input matrix might be

$X = \begin{bmatrix} x_{11} & x_{12} & x_{13} \\ x_{21} & x_{22} & x_{23} \\ x_{31} & x_{32} & x_{33} \\ x_{41} & x_{42} & x_{43} \\ x_{51} & x_{52} & x_{53} \end{bmatrix}$

where $x_{ij}$ is the input in sample $i$ of feature $j$.

What I don't understand

Everywhere I go I see $f(W^TX +\mathbb{b})$ as the formula. However this clearly doesn't make sense here, as we would have incompatible shapes!! $W$ has shapes $(\text{number nodes}, \text{number features})$, while $X$ has shape $($ batch size, number features $)$.

Ideally we would like, for each sample, to have something like this (I only write the first two columns). But how do we get there? Is this the correct way?

$A^{(1)} = \begin{bmatrix} w_{11}^{(1)}x_{11} + w_{12}^{(1)}x_{12} + w_{13}^{(1)}x_{13} & w_{11}^{(1)}x_{21} + w_{12}^{(1)}x_{22} + w_{13}^{(1)}x_{23}\\ w_{21}^{(1)} x_{11} + w_{22}^{(1)}x_{12} + w_{23}^{(1)}x_{13} & w_{11}^{(1)}x_{21} + w_{12}^{(1)}x_{22} + w_{13}^{(1)}x_{23}\\ w_{31}^{(1)}x_{11} + w_{32}^{(1)}x_{12} + w_{33}^{(1)}x_{13} & w_{31}^{(1)}x_{31} + w_{32}^{(1)}x_{32} + w_{33}^{(1)}x_{33}\\ w_{41}^{(1)}x_{11} + w_{42}^{(1)}x_{12} + w_{43}^{(1)}x_{13} & w_{41}^{(1)}x_{41} + w_{42}^{(1)}x_{42} + w_{43}^{(1)}x_{43} \end{bmatrix}$

I don't see the problem. Here's a neural net with three layers: $$\begin{array}{c} y = \beta_0 + \mathbf{V}_1\beta \\ \mathbf{V}_1 = a(\gamma_1 + \mathbf{V}_2\Gamma_1)\\ \mathbf{V}_2 = a(\gamma_2 + \mathbf{V}_3\Gamma_2)\\ \mathbf{V}_3 = a(\gamma_3 + \mathbf{X}\Gamma_3) \end{array}$$

Where $\mathbf{X}$ is your data.

Say that $\mathbf{X}$ is $N \times p$. That means that $\Gamma_3$ is $p \times p_3$, $\mathbf{V}_3$ is therefore $N \times p_3$ and $\Gamma_2$ is $p_3 \times p_2$, $\mathbf{V}_2$ is therefore $N \times p_2$ and $\Gamma_1$ is $p_2 \times p_1$, and $\beta$ is $p_1 \times 1$ (assuming a univariate response).

For a minibatch, just swap out $N$ with $B<N$. It doesn't change the dimensions of the weights, just that of the derived variables/hidden layers $\mathbf{V}$

Your problem is that you were thinking of $X$ as row-oriented vs column oriented, I think. That's why your matrices weren't conformable. But minibatch has nothing to do with that -- minibatch chops off observations, not features. Dropout, on the other hand, chops off features. In so doing it propagates the deleted columns upward/downward on the forward backward passes.

• Inevitably! I am of the opinion that the tensor-based and index based way that neural nets are taught makes them seem needlessly complex, especially to people with a wit of math background. – generic_user Oct 20 '17 at 14:14
• By knowing $X$ is $N \times p$ and $\Gamma_3$ is $p\times p_3$, we know that $V_3$ is $N \times p_3$ – Euler_Salter Oct 20 '17 at 14:22
• IDK but I'm fairly confident that you could use commutative rules for matrices to make this work in row-wise representations of data. – generic_user Oct 20 '17 at 14:25
• Yeah. Editing.. – generic_user Oct 20 '17 at 14:41
• Any tensor can be represented in two dimensions. If you are writing code in assembly or for a GPU, the tensor representation might be optimal. But if you're relying on either blas or sparse matrix libraries, I think the 2d representation is better. But ultimately the two are mathematically equivalent. – generic_user Oct 20 '17 at 16:02