I thought it might be a good exercise to try my hand at making a simple, one hidden layer neural net from scratch. But, for whatever reason, I can't get my in-sample error to go down. I think it has something to do with my backpropagation algorithm being wrong, because the outputs, after many iterations of stochastic gradient descent, tend to move to .33333, meaning that, they're all diverging and it's telling me that there is an equal likelihood of all the possibilities.

The input dimension is 4 and the output dimension is 3 (the Iris flower data in specific, if you know the data set). The output dimension has a softmax activation function, and I'm using a hidden dimension of size 4 with a tanh activation function. My weight set 1 is of size (input dimension x hidden dimension) and my second weights are of size (hidden dimension x output dimension). All weight values are initialized randomly and error is measured using the cross-entropy error. That should be about as much background as is needed I think, but let me know if I forgot something.

Here's some code so you know exactly how I'm performing my backpropagation, but I'll also include my error measure, in the event that maybe I'm just not getting an accurate measure of the error.

Stochastic Gradient Descent:

def StochasticGD(self, numIterations, X, y):

    #storage for the best performing weights in-sample
    bestError = 1000000 #very poor practice. Note to self: fix this       
    bestW1 = np.zeros((self.inputDim, self.hidDim))
    bestb1 = np.zeros((1, self.hidDim))
    bestW2 = np.zeros((self.hidDim, self.outputDim))
    bestb2 = np.zeros((1,self.outputDim))

    for i in range (0,numIterations):
        #select random x_n to perform SGD
        n = np.random.randint(len(y))            
        x_n= X[n,:]
        #for whatever reason, the above line retrurns shape of (,4) instead of (1,4)
        #thus we have to reshape, it seems
        x_n = x_n.reshape((1,4))
        #forward propogation and get probabilities of each
        output,z1 = self.ForProp(x_n)   


        #calculate deltas for each layers
        delta3 = output
        delta3-=1 #derivative of softmax
        delta2 = (delta3.dot(self.W2.T))*(1-(z1**2)) #derivative of tanh * (delta * Weights^T)

        #calculate individual gradients from the deltas
        dW2 = (z1.T).dot(delta3)
        db2 = np.sum(delta3, axis=0, keepdims = True)
        dW1 = (x_n.T).dot(delta2)
        db1 = np.sum(delta2, axis = 0, keepdims=True)

        #add regularizer slowdown to gradients
        dW2 += self.regLambda * self.W2
        dW1 += self.regLambda * self.W1

        #adjust weights
        self.W2-= self.alpha * dW2
        self.b2-= self.alpha * db2
        self.W1-= self.alpha * dW1
        self.b1-= self.alpha * db1

        #reclaculate error
        newOutput,___ = self.ForProp(X)
        error = self.CalculateError(newOutput, y)
        #if error went down, log these weights
            bestError = error
            bestW1 = self.W1
            bestb1 = self.b1
            bestW2 = self.W2
            bestb2 = self.b2

    #after loop take on the best values
    self.W1 = bestW1
    self.b1 = bestb1
    self.W2 = bestW2
    self.b2 = bestb2

And here's my error measure:

def CalculateError(self, output, y):
    #calculate total error against the vector y for the neurons where output = 1 (the rest are 0)    
    totalError = 0
    for i in range(0,len(y)):
        totalError += -np.log(np.array(output)[i, int(y[i])])

    #now account for regularizer
    totalError+=(self.regLambda/self.inputDim) * (np.sum(np.square(self.W1))+np.sum(np.square(self.W2)))     

    error=totalError/len(y) #divide ny N
    return error

Thanks in advance for any help you can provide

UPDATE: I changed the hidden layer size to 3, because apparently having a hidden layer of the same size as your input layer means no learning is happening. That said, my model is still not learning, but the error isn't going up anymore either, so at least there's that. I just thought I'd include that in the event somebody ever stumbles on this, that might be something to try as well.