My NN accuracy is bouncing between .29 and .37. Sometimes it starts at .5, but then decreases as it continues. The loss also bounces around, decreasing, increasing, and generally staying around 1. The goal is to differentiate between user handwriting. The user gives it an image, it takes the image and splits it into letters, then it labels each letter to each user and trains the network with the labels and the letter images as the data.

It, uh, doesn't work very well. I'm a beginner when it comes to Neural Networks, and would really love advice and pointers. We don't have a lot of data, and I know we need a lot more, but I'm still trying to accomplish some kind of learning. Here is the code:

def trainNeuralNetwork(users, data, labels):

    IMAGES = 10

    data = np.array(data, dtype="float32")/ 255.0
    labels = np.array(labels)

    #mlb = preprocessing.MultiLabelBinarizer()
    #transformed_label = mlb.fit_transform(labels)

    lb = preprocessing.LabelBinarizer()
    transformed_label = lb.fit_transform(labels)

    #data = data.transpose()

    #transformed_label = transformed_label.reshape(((len(users))*10),)

    X_train, X_test, y_train, y_test = train_test_split(data, transformed_label, test_size=0.20, random_state=111)

    #y_train = np_utils.to_categorical(y_train, len(users))
    #y_test = np_utils.to_categorical(y_test, len(users))
    learning_rate = 0.005

    momentum = 0.8
    model = Sequential()

    #amountData = len(users) * IMAGES

    model.add(Convolution2D(32, 3, activation='relu', input_shape=(28,28,3)))
    model.add(Convolution2D(32, 3, activation='relu'))


    model.add(Dense(32, activation='relu'))
    model.add(Dense(len(users), activation='softmax'))

    sgd = optimizers.SGD(lr = learning_rate, decay = 1e-6, momentum = momentum, nesterov = False)


    #y_train = y_train[:, 0, :]
    model.fit(X_train, y_train, 
              batch_size=32, nb_epoch=30, verbose=1)
    #y_test = y_test[:, 0, :]
    score = model.evaluate(X_test, y_test, verbose=0)

    model_yaml = model.to_yaml()
    with open("model.yaml", "w") as yaml_file:

users = ["User1", "User2", "User2"]
data, labels, tempwords = LetterBreaker.imageProcess(users, 10)
trainNeuralNetwork(users, data, labels)

What would be causing this error where it bounces between accuracy? I've played round a lot with learning rate, but it seems like no matter the learning rate it does this. The accuracy is incredibly low.