mini batch matrix form of neural network I am studying neural network. I read a lot about mini-batch gradient descent. So now, I was implementing it in TensorFlow.
But I have some questions regarding the matrix formulation of the problem/vectorization.
Notation
First some notation 
$w_{ij}^l$ is the weight on the connection connecting node $j$ in layer $l$ to node $i$ in layer $l+1$. 
If we have $4$ nodes and $3$ features we could have:
$W^{(1)} = \begin{bmatrix}
    w_{11}^{(1)}       & w_{12}^{(1)}    & w_{13}^{(1)}  \\
    w_{21}^{(1)}       & w_{22}^{(1)}    & w_{23}^{(1)}  \\
    w_{31}^{(1)}       & w_{32}^{(1)}    & w_{33}^{(1)}  \\
    w_{41}^{(1)}       & w_{42}^{(1)}    & w_{43}^{(1)}
\end{bmatrix}$
At the same time, suppose we have a batch size of $5$, then the input matrix might be
$X = \begin{bmatrix}
    x_{11}       & x_{12}    & x_{13}  \\
    x_{21}       & x_{22}    & x_{23}  \\
    x_{31}       & x_{32}    & x_{33}  \\
    x_{41}       & x_{42}    & x_{43}   \\
    x_{51}       & x_{52}    & x_{53}
\end{bmatrix}$
where $x_{ij}$ is the input in sample $i$ of feature $j$.
What I don't understand
Everywhere I go I see $f(W^TX +\mathbb{b})$ as the formula. However this clearly doesn't make sense here, as we would have incompatible shapes!! $W$ has shapes $(\text{number nodes}, \text{number features})$, while $X$ has shape  $($ batch size, number features $)$.
Ideally we would like, for each sample, to have something like this (I only write the first two columns). But how do we get there? Is this the correct way?
$A^{(1)} = \begin{bmatrix}
    w_{11}^{(1)}x_{11}       + w_{12}^{(1)}x_{12}   + w_{13}^{(1)}x_{13} & w_{11}^{(1)}x_{21}       + w_{12}^{(1)}x_{22}   + w_{13}^{(1)}x_{23}\\
    w_{21}^{(1)} x_{11}      + w_{22}^{(1)}x_{12}    + w_{23}^{(1)}x_{13}  & w_{11}^{(1)}x_{21}       + w_{12}^{(1)}x_{22}   + w_{13}^{(1)}x_{23}\\
    w_{31}^{(1)}x_{11}       + w_{32}^{(1)}x_{12}    + w_{33}^{(1)}x_{13}  & w_{31}^{(1)}x_{31}       + w_{32}^{(1)}x_{32}    + w_{33}^{(1)}x_{33}\\
    w_{41}^{(1)}x_{11}       + w_{42}^{(1)}x_{12}    + w_{43}^{(1)}x_{13} &  w_{41}^{(1)}x_{41}       + w_{42}^{(1)}x_{42}    + w_{43}^{(1)}x_{43}
\end{bmatrix}$
 A: I don't see the problem.  Here's a neural net with three layers:
$$
\begin{array}{c}
y = \beta_0 + \mathbf{V}_1\beta \\
\mathbf{V}_1 = a(\gamma_1 + \mathbf{V}_2\Gamma_1)\\
\mathbf{V}_2 = a(\gamma_2 + \mathbf{V}_3\Gamma_2)\\
\mathbf{V}_3 = a(\gamma_3 + \mathbf{X}\Gamma_3)
\end{array}
$$
Where $\mathbf{X}$ is your data.  
Say that $\mathbf{X}$ is $N \times p$.  That means that $\Gamma_3$ is $p \times p_3$, $\mathbf{V}_3$ is therefore $N \times p_3$ and $\Gamma_2$ is $p_3 \times p_2$, $\mathbf{V}_2$ is therefore $N \times p_2$ and $\Gamma_1$ is $p_2 \times p_1$, and $\beta$ is $p_1 \times 1$ (assuming a univariate response). 
For a minibatch, just swap out $N$ with $B<N$.  It doesn't change the dimensions of the weights, just that of the derived variables/hidden layers $\mathbf{V}$ 
Your problem is that you were thinking of $X$ as row-oriented vs column oriented, I think.  That's why your matrices weren't conformable.  But minibatch has nothing to do with that -- minibatch chops off observations, not features.  Dropout, on the other hand, chops off features.  In so doing it propagates the deleted columns upward/downward on the forward backward passes.
