0
$\begingroup$

I am trying to program an OCR based on neural networks.

It has in sum 3 layers : Input( 32 * 32 ) => Hidden (180) => Output(2)

It task is to recognize the numbers 1 and 2. If I let the network train for example to figure out what "1" is, I set as target [1, 0]. It backpropagates until the output is less than the treshhold( id est the output of the desired neurone is 0.999 then the treshold is calculated as (1 - output)). If I train now the number 2 ( target [0, 1]) it trains correctly to [1E-7, 0.9] but forgets the old training and therefore tends in the output layer more likely to the number "2".

How can I train the neural network to the effect, that both trainings still effecting the output? Because the training for number "2" is newer, it is more weighted in the output. If I give as input an "1" the output will something like [0.1, 0.2]...

To better understand: Given is a training set of ten examples of number "1" and ten examples of number "2". I first let the network_ train the number 1 with these ten examples until the output ist something like [0.999, 0.001]. The problem is now that whenever I try to train the network for the number 2, the error is extremly big, because the network is trained to recognize the number "1". How can I _let the network train for recognizing "1" AND "2" without losing the information of the last training?

Thanks in advance!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

You need to shuffle your training set randomly, and take only a single step in the reverse direction of the gradient at each iteration. What you have been doing is overfitting to the "1" set, and then overfitting to the "2" set.

Improve this answer
$\endgroup$
4
  • $\begingroup$ So, you mean, I feed forward an example "1" and backpropagate 1 time, and then feed forward an example "2" and backpropagate one time and so on, until all training data is completed? Don't I need to backpropagate each training data until the desired output is learned? Hm.. makes sense $\endgroup$
    – φ Const. NET
    Jan 28, 2018 at 17:19
  • $\begingroup$ @φConst.NET Backpropagate each data once, then move on to the next data. If necessary, after training on all the data, shuffle randomly and feed it one at a time again. Each time you go through all your data is called an epoch. $\endgroup$
    – elliotp
    Jan 28, 2018 at 22:18
  • $\begingroup$ Hello, one more question: Given is a trainig set with just one example for number "1" and another example for number "2". Is it possible to train the network with just two examples? How often do I need to iterate through the sets? Because I tried with at least three examples for each numbers and iterate through these 200 times but still the output is everytime like 0.500192 and 0.500153 _ thanks in advance! $\endgroup$
    – φ Const. NET
    Jan 29, 2018 at 0:24
  • $\begingroup$ If you have just one example, a neural network is not expected to perform any better than a much simpler method such as logistic regression (or softmax for more than 2 classes). In fact, it may perform much worse because it memorizes those two examples, why a simpler model will generalize to examples that is hasn't seen. If you insist on using a neural network, I would alternate between the two examples. You may have to iterate through both of them many thousands of times. It should eventually give you good results, at least for your training set. $\endgroup$
    – elliotp
    Jan 29, 2018 at 5:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.