10
$\begingroup$

Is it valid to train a neural network over and over again with new arriving data (including pruning after each new training)?

I plan to collect data for a period of time, train/cv/test the networ, then again collect new data and train the existing and already trained network with the new data.

The time series prediction setup, e.g. described here, doesn't fit I think, as the input and the output isn't the same data type, i.e. the input features are volume based count of a data stream and the output is a class label, so I cannot just "move the data to the left".

My searches on Google yielded mainly results regarding "incremental vs. batch" training of the weights of a model or incremental growing of the hidden layer. This paper seems to be what I'm looking for, but I'm still not completely confident about the usage of recurrent neural networks.

I could also create a new network for each time period, but thus I'd lose the knowledge gathered from the previous time periods.

So what do you suggest?

$\endgroup$
1
  • $\begingroup$ If you use regularization in the training then you don't need to prune. The big ones are finicky, so getting them to work well with continuous training can be a challenging proposition. $\endgroup$ Nov 12, 2020 at 13:19

2 Answers 2

4
$\begingroup$

I would suggest you to use Transfer Learning Techniques. Basically, it transfers the knowledge in your big and old dataset to your fresh and small dataset.

Try reading: A Survey on Transfer Learning and the algorithm TrAdaBoost.

$\endgroup$
1
$\begingroup$

This branch of ML is referred to as Continual Learning (or Incremental Learning). It is effectively multi-task transfer learning, where the tasks come sequentially, and we wish to use an architecture which does not "catastrophically forget" the information from the previous tasks. See the following paper:

Continual Lifelong Learning with Neural Networks: A Review (2019)

And the following related questions:

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.