I'm building a neural network from scratch that is using basic c# functionality like 2D arrays, loops etc I do not want to use TensorFlow or any other packages because my goal is to move this code to C and embed it into a chip. Also im doing it as a learning process.

I made a supervised perceptron network with 16384 inputs (128x128 digits from the MNIST database) and 1 hidden layer with 128 hidden neurons and 10 outputs. My back propagation is working, im not overfitting and i can get it to work fine on one digit at a time.

My problem is when i want to batch train the network on an image like '2' for 1000 iterations and then i train it to recognise a '5' using the final weights after training the '2' it loses the ability to recognise a 2. I figured its because i have to do random character training and keep track of what im training so the network can slowly adjust to be able to weight everything correctly for numbers 0-9 but this doesnt work either. Any suggestions on how I am supposed to train a network simultaneously?

Here is my code for the batch training. Assume that the feed forward and backprop are working fine. bias for hidden outputs is 0.5 and bias for output is 0.35. Weights are initally randomized between 0.00000 and 1.0000 Thanks!

string[] images = { "C:\\NIST\\BATCH_TRAIN\\0.png",
                                        "C:\\NIST\\BATCH_TRAIN\\9.png"  };

        var watch = System.Diagnostics.Stopwatch.StartNew();
        Random r = new Random();
        int i = 0;
        int count = 0;
        for (int n = 0; n < 10000; n++)
            i = r.Next(0, 10);
            if(i != 10)
                //clear inputs

                //Load image into bitmap
                Bitmap img = new Bitmap(images[i]);
                int pixelCount = img.Width * img.Height;

                //Get the input values
                for (int k = 0; k < img.Width; k++)
                    for (int j = 0; j < img.Height; j++)
                        Color pixel = img.GetPixel(k, j);
                        if (pixel.GetBrightness() > 0.5) //means white
                            //Means black ink pixel found

                //determine and set the target based on the image we are training
                double[] target = { 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01 };
                target[i] = 0.91;

                //Add to dictionary for tracking how many and which images were trained
                d[i] = d[i] + 1;

                //Feed forward

                //Then backpropagate Output Layer to Hidden Neurons

                //Then backpropagate Hidden Neurons to Input Layer
                backpropagateIH(target, eta);

                //Then clear  input list

                //increment counter


        var seconds = watch.ElapsedMilliseconds/1000;
  • $\begingroup$ $$\text{My problem is when i want to batch train the network on an image like '2' for 1000 iterations ...}$$ Why are you doing it this way? This behavior is called "catastrophic forgetting." It arises because you're training on the same data over and over, and the network "forgets" what else it is supposed to be learning. Shuffle your data, and increase your minibatch size. $\endgroup$ – Sycorax says Reinstate Monica Jul 26 '18 at 1:44
  • $\begingroup$ Good point i dont know why i was doing that. I was trying to do generative training but i completely forgot to apply transformations, rotations and noise to my data so it was training the same thing over. I will try that $\endgroup$ – i_shoot_photos Jul 26 '18 at 12:28

Probably calculate for each image the entropy, energy, contrast, homogeneity, based on the co-occurrence matrix of the delta-matrix of gray-scaled pixel values ($R=G=B=0,1,\ldots,255$). Also, after gray-scaling each image, just line up the pixel values into a vector and calculate the 3rd and 4th moments of the vector. In addition, calculate the "Hu moments" (M. Hu. Visual pattern recognition by moment invariants. IRE Trans. Inf. Theor. IT-8: 179-187, 1962). When done, for each image, clamp the above values to the input side of the ANN and don't input what you are currently using. You basically have to perform a lot of shape transformations on the images, and then use those values as input feature values as inputs to the ANN.

While you are getting started with the ANN, I wouldn't say you are not overfitting, as a regularization fanatic would destroy you in a review or via audience participation. So start now, and look at the newer "dropout" method of ANN regularization, whereby you randomly drop out (e.g., zero the activation function values) of maybe 20-50% of the hidden nodes, so the error updates percolating back to the connection weights during back-propagation learning don't get burned in during sweeps (epochs). Dropout is a newer contemporary method that has been proposed as an alternative to the Mackay-type Bayesian regularization. FYI - you can still overfit after Bayesian regularization. Dropout is faster, since you don't have to work with the trace of the Hessian matrix during every iteration.

  • $\begingroup$ Thank you for your detailed suggestion. DoBy dropping out/zero-ing the activation is that similar to 'Pruning Networks'? I will give it a try and see. Maybe I am overfitting too. $\endgroup$ – i_shoot_photos Jul 25 '18 at 19:25
  • 1
    $\begingroup$ yes, it's a type of pruning. However, "pruning" terminology is not affiliated with the newer dropout approach. For example, there is also "boosting," which is not related to dropout. $\endgroup$ – JoleT Jul 25 '18 at 19:41
  • $\begingroup$ i just want to let you know i figured out my weight set was bad. After correcting it (i was using initial random weights that were too small so most of my hidden firiing very small values) i can now draw a number and have it recognise it with about 80% accuracy. Now I think i will look into what you suggested and use that to optimize my network. Thanks again! $\endgroup$ – i_shoot_photos Aug 3 '18 at 17:26

Not the answer you're looking for? Browse other questions tagged or ask your own question.