I thought it might be a good exercise to try my hand at making a simple, one hidden layer neural net from scratch. But, for whatever reason, I can't get my in-sample error to go down. I think it has something to do with my backpropagation algorithm being wrong, because the outputs, after many iterations of stochastic gradient descent, tend to move to .33333, meaning that, they're all diverging and it's telling me that there is an equal likelihood of all the possibilities.
The input dimension is 4 and the output dimension is 3 (the Iris flower data in specific, if you know the data set). The output dimension has a softmax activation function, and I'm using a hidden dimension of size 4 with a tanh activation function. My weight set 1 is of size (input dimension x hidden dimension) and my second weights are of size (hidden dimension x output dimension). All weight values are initialized randomly and error is measured using the cross-entropy error. That should be about as much background as is needed I think, but let me know if I forgot something.
Here's some code so you know exactly how I'm performing my backpropagation, but I'll also include my error measure, in the event that maybe I'm just not getting an accurate measure of the error.
Stochastic Gradient Descent:
def StochasticGD(self, numIterations, X, y): #storage for the best performing weights in-sample bestError = 1000000 #very poor practice. Note to self: fix this bestW1 = np.zeros((self.inputDim, self.hidDim)) bestb1 = np.zeros((1, self.hidDim)) bestW2 = np.zeros((self.hidDim, self.outputDim)) bestb2 = np.zeros((1,self.outputDim)) for i in range (0,numIterations): #select random x_n to perform SGD n = np.random.randint(len(y)) x_n= X[n,:] #for whatever reason, the above line retrurns shape of (,4) instead of (1,4) #thus we have to reshape, it seems x_n = x_n.reshape((1,4)) #forward propogation and get probabilities of each output,z1 = self.ForProp(x_n) #backprop #calculate deltas for each layers delta3 = output delta3-=1 #derivative of softmax delta2 = (delta3.dot(self.W2.T))*(1-(z1**2)) #derivative of tanh * (delta * Weights^T) #calculate individual gradients from the deltas dW2 = (z1.T).dot(delta3) db2 = np.sum(delta3, axis=0, keepdims = True) dW1 = (x_n.T).dot(delta2) db1 = np.sum(delta2, axis = 0, keepdims=True) #add regularizer slowdown to gradients dW2 += self.regLambda * self.W2 dW1 += self.regLambda * self.W1 #adjust weights self.W2-= self.alpha * dW2 self.b2-= self.alpha * db2 self.W1-= self.alpha * dW1 self.b1-= self.alpha * db1 #reclaculate error newOutput,___ = self.ForProp(X) error = self.CalculateError(newOutput, y) #if error went down, log these weights if(error<bestError): bestError = error bestW1 = self.W1 bestb1 = self.b1 bestW2 = self.W2 bestb2 = self.b2 #after loop take on the best values self.W1 = bestW1 self.b1 = bestb1 self.W2 = bestW2 self.b2 = bestb2
And here's my error measure:
def CalculateError(self, output, y): #calculate total error against the vector y for the neurons where output = 1 (the rest are 0) totalError = 0 for i in range(0,len(y)): totalError += -np.log(np.array(output)[i, int(y[i])]) #now account for regularizer totalError+=(self.regLambda/self.inputDim) * (np.sum(np.square(self.W1))+np.sum(np.square(self.W2))) error=totalError/len(y) #divide ny N return error
Thanks in advance for any help you can provide
UPDATE: I changed the hidden layer size to 3, because apparently having a hidden layer of the same size as your input layer means no learning is happening. That said, my model is still not learning, but the error isn't going up anymore either, so at least there's that. I just thought I'd include that in the event somebody ever stumbles on this, that might be something to try as well.