A Neural Network is essentially a set of weight matrices. Let's call them H1, H2, H3 etc (where each index is the hidden layer number).
In the forward pass, we take the input batch/dataset as a matrix X, which we then multiply with the first matrix H1 (after adding bias), and then H2 (after adding bias) and so on to get the output.
Could a speedup be achieved by multiplying the matrices out of order, before the result is multiplied with the input X? Matrix Chain Optimization could be used to reduce the raw number of computations. Also, the matrices could be multiplied in parallel.