3
$\begingroup$

I'm trying to build my own neural network class (I know there's 1001 already out there, but this is how I learn best). I'm having a bit of trouble with the sizes of certain vectors. For this I'm working off the following source: Neural Networks.

My problem is calculating the z vector (z is the weighted sum of inputs calculated for each neuron in each layer).

z(2) = w(1) * x + b

Where:

w = weight matrix of size (m x n) where m = the number of neuron in the input layer, and n = the number of neuron in second layer
x = input matrix of size (s x t) where s is the number of neuron in the input layer and t is the number of training examples.
b = input matrix of size (c x d) where c is the number of neurons in the second layer and d is the number of neurons in the input layer

This equation changes for the next layer, but that's not something I'm worried about atm. I know that matrix b is essentially just a vector that's been copied across the width.

Now there's a couple of things I can state a fact from the above: m = s = d, n = c. However when I try and process the above equation none of my matrix sizes match.

Can someone tell me given the list of knowns below what the sizes of the matrix z, w, x and b should be.

1. Input size (number of neurons)
2. Number of neurons in layer 2
3. Number of training examples

I've attempted this sort of thing before and I think I always crash and burn when it comes to initialising the arrays (I'm actually using jagged arrays, but they are rectangular).

$\endgroup$
4
$\begingroup$

You say

"w = weight matrix of size (m x n) where m = the number of neuron in the input layer, and n = the number of neuron in second layer"

You should think in terms of "out = func(in)", so "$m$ in, $n$ out", and "$x$ in, $z$ out".

Now you want "out = matrix * in", so the way matrix multiplication works you will need $$n = [n,m] * m$$ i.e. the rule is that adjacent dimensions must be the same in a multiplication formula.

So something seems off. Indeed, if you look at the first equation in your link, it is the transpose of the weight matrix that is used $$z=W^Tx$$ This makes sense, because $[m,n]^T=[n,m]$, as required.

$\endgroup$
  • $\begingroup$ you haven't answered it.. $\endgroup$ – Euler_Salter Oct 20 '17 at 9:37
0
$\begingroup$

Chuck Anderson does a great job of explaining these details here:

http://nbviewer.jupyter.org/url/www.cs.colostate.edu/~anderson/cs480/notebooks/10%20Nonlinear%20Regression%20with%20Neural%20Networks.ipynb

The associated video is also very good. HTH

$\endgroup$
  • $\begingroup$ dead link. can you please update the link $\endgroup$ – Eswar Yaganti Mar 2 '18 at 0:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.