0
$\begingroup$

Let's suppose I have a neural network with one hidden layer. During training, for a given pair of (input, target), I want to perform several iterations, such that the first iteration would be trying to predict the target, and the second iteration would be to somehow use my prediction (or other information learned from the first iteration) as my new target.

My initial thinking to solve this would be to go through a full epoch using the initial true targets and then at the second epoch, I would be able to use the predictions as the new targets. However, this seems like it could all be integrated in one network, end to end.

Is there a possible way to do this without having some information leakage?

Just for people interested in something similar, I found this blog post

$\endgroup$
  • $\begingroup$ What problem are you trying to solve by using this method? What benefits are you trying to obtain using this technique? $\endgroup$ – Sycorax Apr 7 at 16:42
  • $\begingroup$ This question is slightly related to:stats.stackexchange.com/questions/401576/…. In a sense, I would like the input to have more effect than my label when training a model. Assume that for a given pair (input,label), the input is responsible for 50% when predicting the label and the label itself is responsible for 50% as well. If I then use my prediction as the new target, I again have 50% coming from the same input and 50% coming from my new target (but that target is already 50% influenced by the input) $\endgroup$ – Tom Apr 7 at 17:18
  • $\begingroup$ So if I just re-use my prediction once as the target, I get that I have 50% from the input (on the second iteration) + 50% of 50% = 25% from the first iteration (using the original target), if that makes sense. $\endgroup$ – Tom Apr 7 at 17:19
  • 1
    $\begingroup$ i don't understand -- you want to predict the output of the network? $\endgroup$ – shimao Apr 9 at 21:19
  • $\begingroup$ Yes, for the same inputs. The problem is that, to do so, I would need to have already computed it and then explicitly use the output of the network as my new target. I am wondering whether it is possible to have it integrated in the same network architecture. $\endgroup$ – Tom Apr 10 at 0:11
0
$\begingroup$

https://wayve.ai/blog/dreaming-about-driving-imagination-rl is describing a single network. They are not describing a procedure where the prediction is used as input for further prediction.

the data they have is a series of observed driving conditions lets call it $$DC={DC_1,DC_2,DC_3,...DC_N}$$

Note their statement:

We train the encoder and prediction model on real-world data.

At no point are predictions used as inputs to train the neural network.

Training is performed in this fashion

DC_K -> [neural network] -> [prediction] ------|
                                               |->[reward to neural network]
DC_{K+1} --------------------------------------|

the prediction is not being used as input into the neural network it is only being used as comparison with DC_{K+1} which is being used as the input to the neural network.

DC_K -> [neural network] -> [prediction] ------|
                                               |->[reward to neural network]
DC_{K+1} --------------------------------------|
   |
   ---> [neural network] -> [prediction] ------|
                                               |->[reward to neural network]
DC_{K+2} --------------------------------------|
   |
   ---> [neural network] -> [prediction] ------|
                                               |->[reward to neural network]
DC_{K+3} --------------------------------------|
   |
  ...
   |
   ---> [neural network] -> [prediction] ------|
                                               |->[reward to neural network]
DC_{N} ----------------------------------------|

notice that at no point are the predictions being used as inputs to the neural network, this is not being done in a sequence of training operations as might be inferred from their diagrams but rather the rewards are all applied in a single stage as would be expected from a normal stage of neural network training. The only reason they are making it look like the outputs are being used as inputs is to explain the serial nature of their training data, that DC_{K} is both input and target.

In the explanation for training their driving policy they do use the prediction output as the input to the next stage of prediction. However at this point the prediction model is not being trained, and the output of the driving policy is just steering and speed information. The paper leaves out how they calculate the reward given to the policy network based on the predictions. I would assume they have another network that has been trained to evaluate how "off the road" a given state vector is.

Another, less likely option, is that they just take the difference of the predicted state vectors vs actual state vectors given for each possible DC_1. Since if DC is based on proper driving behavior then each predicted DC_k should match the original real world data if the driving policy is matching the proper driving behavior. I doubt they are doing this.

$\endgroup$
  • $\begingroup$ Thank you for your answer. However, I was not looking to use the predictions as new inputs but rather as new targets. I think in their case they have an autoregressive model, where as you pointed out, states are both inputs and outputs. $\endgroup$ – Tom Apr 12 at 11:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.