I implemented Convolutional Neural Network from scratch for image recognition for 5 classes. When I train it using only one image from one class it seems to be working, because accuracy for this class increase while decrease for others. However when I try to train it for every class, the accuracy for every class keep changing but it still remains nearly the same for every class.
The way I feed image to CNN is that I take few images for every class and feed it to CNN. After all computations I update the weights.
My cnn structure is:
IMAGE(64x64x3) -> CONV (5 filter 5x5x3) -> ReLU -> Pool -> CONV (10 filter 5x5x10) -> ReLU -> Pool -> FCL(1690) -> FCL(500) -> FCL(120) -> Softmax(5)
What am I doing wrong?