I am looking at an example for the activation function $a_1$.

Why does the equation look like $\Theta_{10}x_0 + \Theta_{11}x_1 + \Theta_{12}x_2 + \Theta_{13}x_3$

instead of

$\Theta_{10}x_0 + \Theta_{11}x_1 + \Theta_{21}x_2 + \Theta_{31}x_3$?

My understanding is that all input nodes will contribute to the computation of activation value of $a_1$. And hence, shouldn't $\Theta_{xy}$ be the value from $x$ to $y$? Which in this case of $\Theta_{21}$ be from Node $2$ in first layer to Node $1$ in second layer. Am I misunderstanding something?

This is the url http://www.holehouse.org/mlclass/08_Neural_Networks_Representation_files/Image%20[5].png


1 Answer 1


To quote the very page the image is from:


  • $j$ (first of two subscript numbers)= ranges from $1$ to the number of units in layer $l+1$
  • $i$ (second of two subscript numbers) = ranges from $0$ to the number of units in layer $l$
  • $l$ is the layer you're moving FROM

I guess the choice for this is to be able to write it with matrices:

$ \underbrace{ \begin{pmatrix} \Theta_{00} & \Theta_{01} & \Theta_{02} & \Theta_{03} \\ \Theta_{10} & \Theta_{11} & \Theta_{12} & \Theta_{13} \\ \Theta_{20} & \Theta_{21} & \Theta_{22} & \Theta_{23} \\ \end{pmatrix} }_{\Huge\Theta_{ji}} * \begin{pmatrix} x_{0} \\ x_{1} \\ x_{2} \\ x_{3} \end{pmatrix} = \begin{pmatrix} \Theta_{00}x_0 & \Theta_{01}x_1 & \Theta_{02}x_2 & \Theta_{03}x_3 \\ \Theta_{10}x_0 & \Theta_{11}x_1 & \Theta_{12}x_2 & \Theta_{13}x_3 \\ \Theta_{20}x_0 & \Theta_{21}x_1 & \Theta_{22}x_2 & \Theta_{23}x_3 \\ \end{pmatrix} \begin{matrix} (\text{input }a_1) \\ (\text{input }a_2) \\ (\text{input }a_3) \\ \end{matrix} $

$x_0$ is bias by the way.

With $\Theta_{ij}$ you'd have $x^T\Theta_{ij}$ where every column $c$ would be input to $a_c$ but its custom to have the vector after the matrix.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.