1
$\begingroup$

A Neural Network is essentially a set of weight matrices. Let's call them H1, H2, H3 etc (where each index is the hidden layer number).

In the forward pass, we take the input batch/dataset as a matrix X, which we then multiply with the first matrix H1 (after adding bias), and then H2 (after adding bias) and so on to get the output.

Could a speedup be achieved by multiplying the matrices out of order, before the result is multiplied with the input X? Matrix Chain Optimization could be used to reduce the raw number of computations. Also, the matrices could be multiplied in parallel.

$\endgroup$
0
$\begingroup$

Were this possible, it would be a linear transformation and could be represented by a single matrix. There are however nonlinear activation functions. The very first activation function has to be applied to X transformed by the first parameters. Information therefore does literally flow through each step of the network, it's not a single transformation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.