How to pass training data to a neural network for digit recognition [closed]

Question

I'm building a neural network from scratch that is using basic c# functionality like 2D arrays, loops etc I do not want to use TensorFlow or any other packages because my goal is to move this code to C and embed it into a chip. Also im doing it as a learning process.

I made a supervised perceptron network with 16384 inputs (128x128 digits from the MNIST database) and 1 hidden layer with 128 hidden neurons and 10 outputs. My back propagation is working, im not overfitting and i can get it to work fine on one digit at a time.

My problem is when i want to batch train the network on an image like '2' for 1000 iterations and then i train it to recognise a '5' using the final weights after training the '2' it loses the ability to recognise a 2. I figured its because i have to do random character training and keep track of what im training so the network can slowly adjust to be able to weight everything correctly for numbers 0-9 but this doesnt work either. Any suggestions on how I am supposed to train a network simultaneously?

Here is my code for the batch training. Assume that the feed forward and backprop are working fine. bias for hidden outputs is 0.5 and bias for output is 0.35. Weights are initally randomized between 0.00000 and 1.0000 Thanks!

string[] images = { "C:\\NIST\\BATCH_TRAIN\\0.png",
                                        "C:\\NIST\\BATCH_TRAIN\\1.png",
                                        "C:\\NIST\\BATCH_TRAIN\\2.png",
                                        "C:\\NIST\\BATCH_TRAIN\\3.png",
                                        "C:\\NIST\\BATCH_TRAIN\\4.png",
                                        "C:\\NIST\\BATCH_TRAIN\\5.png",
                                        "C:\\NIST\\BATCH_TRAIN\\6.png",
                                        "C:\\NIST\\BATCH_TRAIN\\7.png",
                                        "C:\\NIST\\BATCH_TRAIN\\8.png",
                                        "C:\\NIST\\BATCH_TRAIN\\9.png"  };

        var watch = System.Diagnostics.Stopwatch.StartNew();
        Random r = new Random();
        int i = 0;
        int count = 0;
        for (int n = 0; n < 10000; n++)
        {
            i = r.Next(0, 10);
            if(i != 10)
            {
                //clear inputs
                inputs.Clear();

                //Load image into bitmap
                Bitmap img = new Bitmap(images[i]);
                int pixelCount = img.Width * img.Height;

                //Get the input values
                for (int k = 0; k < img.Width; k++)
                {
                    for (int j = 0; j < img.Height; j++)
                    {
                        Color pixel = img.GetPixel(k, j);
                        if (pixel.GetBrightness() > 0.5) //means white
                        {
                            inputs.Add(0);
                        }
                        else
                        {
                            //Means black ink pixel found
                            inputs.Add(1);
                        }
                    }
                }

                //determine and set the target based on the image we are training
                double[] target = { 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01 };
                target[i] = 0.91;

                //Add to dictionary for tracking how many and which images were trained
                d[i] = d[i] + 1;


                //Feed forward
                feedforward();

                //Then backpropagate Output Layer to Hidden Neurons
                backpropagateOH(target);

                //Then backpropagate Hidden Neurons to Input Layer
                backpropagateIH(target, eta);

                //Then clear  input list
                inputs.Clear();

                //increment counter
                count++;

            }
        }

        watch.Stop();
        var seconds = watch.ElapsedMilliseconds/1000;

Answer 1

Probably calculate for each image the entropy, energy, contrast, homogeneity, based on the co-occurrence matrix of the delta-matrix of gray-scaled pixel values ($R=G=B=0,1,\ldots,255$). Also, after gray-scaling each image, just line up the pixel values into a vector and calculate the 3rd and 4th moments of the vector. In addition, calculate the "Hu moments" (M. Hu. Visual pattern recognition by moment invariants. IRE Trans. Inf. Theor. IT-8: 179-187, 1962). When done, for each image, clamp the above values to the input side of the ANN and don't input what you are currently using. You basically have to perform a lot of shape transformations on the images, and then use those values as input feature values as inputs to the ANN.

While you are getting started with the ANN, I wouldn't say you are not overfitting, as a regularization fanatic would destroy you in a review or via audience participation. So start now, and look at the newer "dropout" method of ANN regularization, whereby you randomly drop out (e.g., zero the activation function values) of maybe 20-50% of the hidden nodes, so the error updates percolating back to the connection weights during back-propagation learning don't get burned in during sweeps (epochs). Dropout is a newer contemporary method that has been proposed as an alternative to the Mackay-type Bayesian regularization. FYI - you can still overfit after Bayesian regularization. Dropout is faster, since you don't have to work with the trace of the Hessian matrix during every iteration.