# Simple Neural Network Issue in Python? [closed]

I thought it might be a good exercise to try my hand at making a simple, one hidden layer neural net from scratch. But, for whatever reason, I can't get my in-sample error to go down. I think it has something to do with my backpropagation algorithm being wrong, because the outputs, after many iterations of stochastic gradient descent, tend to move to .33333, meaning that, they're all diverging and it's telling me that there is an equal likelihood of all the possibilities.

The input dimension is 4 and the output dimension is 3 (the Iris flower data in specific, if you know the data set). The output dimension has a softmax activation function, and I'm using a hidden dimension of size 4 with a tanh activation function. My weight set 1 is of size (input dimension x hidden dimension) and my second weights are of size (hidden dimension x output dimension). All weight values are initialized randomly and error is measured using the cross-entropy error. That should be about as much background as is needed I think, but let me know if I forgot something.

Here's some code so you know exactly how I'm performing my backpropagation, but I'll also include my error measure, in the event that maybe I'm just not getting an accurate measure of the error.

def StochasticGD(self, numIterations, X, y):

#storage for the best performing weights in-sample
bestError = 1000000 #very poor practice. Note to self: fix this
bestW1 = np.zeros((self.inputDim, self.hidDim))
bestb1 = np.zeros((1, self.hidDim))
bestW2 = np.zeros((self.hidDim, self.outputDim))
bestb2 = np.zeros((1,self.outputDim))

for i in range (0,numIterations):
#select random x_n to perform SGD
n = np.random.randint(len(y))
x_n= X[n,:]
#for whatever reason, the above line retrurns shape of (,4) instead of (1,4)
#thus we have to reshape, it seems
x_n = x_n.reshape((1,4))
#forward propogation and get probabilities of each
output,z1 = self.ForProp(x_n)

#backprop

#calculate deltas for each layers
delta3 = output
delta3-=1 #derivative of softmax
delta2 = (delta3.dot(self.W2.T))*(1-(z1**2)) #derivative of tanh * (delta * Weights^T)

#calculate individual gradients from the deltas
dW2 = (z1.T).dot(delta3)
db2 = np.sum(delta3, axis=0, keepdims = True)
dW1 = (x_n.T).dot(delta2)
db1 = np.sum(delta2, axis = 0, keepdims=True)

dW2 += self.regLambda * self.W2
dW1 += self.regLambda * self.W1

self.W2-= self.alpha * dW2
self.b2-= self.alpha * db2
self.W1-= self.alpha * dW1
self.b1-= self.alpha * db1

#reclaculate error
newOutput,___ = self.ForProp(X)
error = self.CalculateError(newOutput, y)
#if error went down, log these weights
if(error<bestError):
bestError = error
bestW1 = self.W1
bestb1 = self.b1
bestW2 = self.W2
bestb2 = self.b2

#after loop take on the best values
self.W1 = bestW1
self.b1 = bestb1
self.W2 = bestW2
self.b2 = bestb2


And here's my error measure:

def CalculateError(self, output, y):
#calculate total error against the vector y for the neurons where output = 1 (the rest are 0)
totalError = 0
for i in range(0,len(y)):
totalError += -np.log(np.array(output)[i, int(y[i])])

#now account for regularizer
totalError+=(self.regLambda/self.inputDim) * (np.sum(np.square(self.W1))+np.sum(np.square(self.W2)))

error=totalError/len(y) #divide ny N
return error