0
$\begingroup$

I've built an ANN from scratch, that works with one row of data with any number of neurons and hidden layers. For the setup I am using 2 hidden layers, 5 neurons (just while building). The network works when I supply it one row of data (with 4 components) and the categorical variable, however I am struggling to get my head around how the structure changes when I want to feed in multiple rows, $x$, each with $n$ components.

I am using numpy library in Python to implement the network. My variables and inputs are as follows:

self.input = np.array([[1,2,3,4],[3,5,1,2],[5,6,7,8]])
self.y = np.array([1,0,1]) #true y
weights_to_hidden = np.random.rand(self.input.shape[1], self.neurons) #(4,5)
self.weights = [(self.neurons,self.neurons),(self.neurons,self.neurons)] 
#one weight matrix for each hidden layer, in my case 2, that is a square matrix of size neurons, in an array
self.weights_to_output = np.random.rand(self.neurons, self.input.shape[0]) #(5, 3)

self.input.shape[0] = number of inputs

self.input.shape[1] = components of each input

self.neurons = 5

I am having issues understanding what sizes my matrices should have, because in its current state I end up with a $(3,3)$ or $(3,4)$ matrix as the output, rather than $(1,3)$ or $(3,1)$. Ideally, I should be able to feed in $x$ rows, each with $n$ components, and get $x$ outputs.

Let's denote

weights from input -> first hidden = $W_{h1}$,

weights from hidden 1 -> hidden2 = $W_{h2}$ and

weights from hidden2 -> output = $W_{o}$.

The activation function is sigmoid, noted $\sigma(x)$. Input is $[x]$ and the matrix of y values at each hidden layer is $[y]_{hn}$. The process of dot product is noted with $*$ operator.

Then, the process of one feedforward is:

  1. $[y]_{h1} = \sigma([x]*W_{h1})$
  2. $[y]_{h2} = \sigma([y]_{h1} * W_{h2})$
  3. out = $\sigma([y]_{h2} * W_{o})$

These operations work, but for back propagation to also be compatible, I need to have one individual value for each input and I am not sure what to change in this procedure for it to be that way. This part of my linear algebra is a bit shaky so any help on this would be very much appreciated!

$\endgroup$
1
$\begingroup$

If the weights are independent for each input, you need an extra dimension, i.e. a 3D tensor of 2D matrices.

If your current matrix is (1,3), then you can pretend it's actually an array of matrices with only one element, i.e. shaped (1,3,1), right? And so if you want x values, well, it'll be (1,3,x).

That's the general idea - don't trust the exact order of indexing I'm using though, I always forget the numpy conventions for that.

$\endgroup$
  • $\begingroup$ I'm not too sure I understand what you mean $\endgroup$ – peterxz Dec 2 '19 at 15:48
  • $\begingroup$ You have an array of arrays. What you need is an array of arrays of arrays., i.e. np.array([ [ [feature1_1, feature1_2], [feature2_1, feature2_2] ], [[feature1_1, feature1_2], [feature2_1, feature2_2] ] ]) $\endgroup$ – jkm Dec 2 '19 at 16:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.