1
$\begingroup$

I have a sequence (> 100 million) of symbols and several models predict the next symbol. To combine these predictions I'm using stacked generalization with a multilayer perceptron trained with online gradient descent.

For the inputs the network uses the predictions of the models so the total inputs are nmodels * nsymbols. As outputs there is one node per symbol. The network is trained on each new symbol.

The network doesn't stabilize, that is, if it stops being trained the predictions become worse and the weights are constantly changing. But the predictions are better than any single model and better than any simple combination such as weighted majority, exponential moving average etc.

I've also tried using as inputs the last N predictions(N * nmodels * nsymbols), the last N predictions + the last N symbols of the sequence, the last N predictions + the last N symbols + the last N probabilities the network assigned for the actual symbol. In all cases the predictions improve but still no stabilization.

The sequence has some times long sub-sequences where the predictions of the models are very accurate but most of the time this doesn't happen.

My main concern is with understanding why this happens. Any ideas?

EDIT: The sequence appears to be non-stationary.

$\endgroup$
  • $\begingroup$ Have you tried training longer? Making the learning rate smaller? Using different batch size? Using different optimizer? $\endgroup$ – Tim Mar 10 at 20:59
  • $\begingroup$ Yes to all, also more hidden nodes, more layers etc. $\endgroup$ – Milton Silva Mar 10 at 21:01
  • $\begingroup$ I don't fully understand what you are doing. What is your network supposed to do? What are the inputs, or rather what do the inputs mean? What do you mean by model? And finally, what are the dimensions of your problem: how many symbols are there? Are you trying to predict the next symbol? Is there anything to pick up in the (training) sequence, or are you trying to guess a random process? $\endgroup$ – cherub Mar 11 at 14:57
  • $\begingroup$ @cherub The network is supposed to learn how to mix the predictions of each model/expert. The number of symbols varies between 4 and 20. There is a sequence of symbols and each expert assigns probabilities from 0 to 1 to each symbol such that they sum to 1. The inputs to the network are those probabilities (normalized). And the output is the probability of each symbol. $\endgroup$ – Milton Silva Mar 11 at 22:33
  • $\begingroup$ Are the experts fixed or do they continue learning as well? $\endgroup$ – christoph Mar 20 at 13:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.