Is Weight Matrix Wij from neuron j to i or from i to j? I have a question about the Weight matrix from a Neural network. I see different notations for this.
I see some publications (e.g. This stanford page) with:
Notation 1:

$W^{(l)}_{ij}$ denotes the parameter (or weight) associated with the
  connection between unit j in layer l, and unit i in layer l + 1.

However in others I read (e.g. This one):
Notation 2:

We denote by $W_{ij}$ the weight of the connection from unit $u_i$ to
  unit $u_j$

So this the other way around. 
My questions:


*

*What is the most common notation?

*For denoting node i to j does $W_{ij}$ has some advantages/disadvantages over $W_{ji}$ ? 

*If I want to vectorize the input vector x multiplied by W (to calculate the input values for the activation function) do I save a transpose so $\textbf{Wx}$ instead of $\textbf{Wx}^{t}$ if I use one or another Weight notation? I am pretty sure the motivation to use $W_{ji}$ is saves 1 transpose on x? Is this correct?


Thanks
 A: • Among the few dozen books and research papers I've read on neural nets  since the late '90s (such as Patrick K. Simpson's excellent cookbook, "Artificial Neural Systems: Foundations, Paradigms, Applications and Implementations"), the most common notation has been to use $Wij$ to denote forward connections from the first layer to the second, at least in two-layer networks. One catch is that some sources label weight matrices in reverse alphabetical order as the number of layers increases; for example, in a three-layer net, the connections from the first layer to the second might be labeled $Vij$. Note that in these cases the alphabetical order of the subscripts remains the same.
• $Wji$ usually represents backwards connections from the second to the first layer, at least in the sources I've slogged through; perhaps it's not always true, but it's not typically used as an alternate designation of $Wij$. I've implemented some of them this way and they've worked fine. The reverse order of the subscripts is usually a dead giveaway for a backwards connection from some later layer to some earlier one. This can get complicated quite quickly though in cases with higher-order connections, ragged connection matrices, connections that skip levels (say from layer three to one) and other special cases where you might see strange symbols like $Vjki$ and the like.
• I usually use SQL sets rather than matrices to implement neural nets, so I can't answer the third part of the question definitively. It may be depend on if $Wij$ and $Wji$ are symmetric or represented by two separate one-way connections that go in opposite directions to the same neurons. Either way, you'd probably have to flip something around, probably using a transpose; I get the same effect in SQL using inner joins between two or more layers. 
I wonder if the notation on the Stanford page might be a mistake, because s$i$, a$i$ and x$i$ are used there to denote the number of nodes, activation values and input numbers for layer L, not layer L + 1. On the other hand, given the multiplicity of different notations for neural nets, it may have some practical significance. This smorgasbord of notations is a real problem in this field and can make implementation really tricky. One way I've found of dealing with this is to find formulas with conflicting notations for the same type of neural net and look for commonalities when implementing them; sometimes the meanings intended by the authors then become clearer. I hope this helps.
