Return to Question

In a recurrent neural network, you would usually forward propagate through several time steps, "unroll" the network, and then back propagate across the sequence of inputs.

Why would you not just update the weights after each individual step in the sequence? (the equivalent of using a truncation length of 1, so there is nothing to unroll) This completely eliminates the vanishing gradient problem, greatly simplifies the algorithm, would probably reduce the chances of getting stuck in local minima, and most importantly seems to work fine. I trained a model this way to generate text and the results seemed comparable to results I have seen from BPTT trained models. I am only confused on this because every tutorial on RNNs I have seen says to use BPTT, almost as if it is required for proper learning, which is not the case.

My only guess for why it is normally propagated through time is that maybe it learns faster or tends to converge on a more accurate solution. I have not yet tried a more traditional RNN with BPTT to get a truly conclusive comparison, but I can't think of any reason why it would be better.

Update: I added an answer

Update: here is the loss function I am using. networkOutputs is the number of different classes to choose from.

        for (int n = 0; n < batchSize; n++)
            for (int i = 0; i < networkOutputs; i++)
                cost -= Math.Log(output[n][i]) * targets[n][i];

        cost /= batchSize;

In a recurrent neural network, you would usually forward propagate through several time steps, "unroll" the network, and then back propagate across the sequence of inputs.

Why would you not just update the weights after each individual step in the sequence? (the equivalent of using a truncation length of 1, so there is nothing to unroll) This completely eliminates the vanishing gradient problem, greatly simplifies the algorithm, would probably reduce the chances of getting stuck in local minima, and most importantly seems to work fine. I trained a model this way to generate text and the results seemed comparable to results I have seen from BPTT trained models. I am only confused on this because every tutorial on RNNs I have seen says to use BPTT, almost as if it is required for proper learning, which is not the case.

Update: I added an answer

In a recurrent neural network, you would usually forward propagate through several time steps, "unroll" the network, and then back propagate across the sequence of inputs.

Why would you not just update the weights after each individual step in the sequence? (the equivalent of using a truncation length of 1, so there is nothing to unroll) This completely eliminates the vanishing gradient problem, greatly simplifies the algorithm, would probably reduce the chances of getting stuck in local minima, and most importantly seems to work fine. I trained a model this way to generate text and the results seemed comparable to results I have seen from BPTT trained models. I am only confused on this because every tutorial on RNNs I have seen says to use BPTT, almost as if it is required for proper learning, which is not the case.

My only guess for why it is normally propagated through time is that maybe it learns faster or tends to converge on a more accurate solution. I have not yet tried a more traditional RNN with BPTT to get a truly conclusive comparison, but I can't think of any reason why it would be better.

Update: I added an answer

Update: here is the loss function I am using

        for (int n = 0; n < batchSize; n++)
            for (int i = 0; i < networkOutputs; i++)
                cost -= Math.Log(output[n][i]) * targets[n][i];

        cost /= batchSize;

In a recurrent neural network, you would usually forward propagate through several time steps, "unroll" the network, and then back propagate across the sequence of inputs.

Why would you not just update the weights after each individual step in the sequence? (the equivalent of using a truncation length of 1, so there is nothing to unroll) This completely eliminates the vanishing gradient problem, greatly simplifies the algorithm, would probably reduce the chances of getting stuck in local minima, and most importantly seems to work fine. I trained a model this way to generate text and the results seemed comparable to results I have seen from BPTT trained models. I am only confused on this because every tutorial on RNNs I have seen says to use BPTT, almost as if it is required for proper learning, which is not the case.

My only guess for why it is normally propagated through time is that maybe it learns faster or tends to converge on a more accurate solution. I have not yet tried a more traditional RNN with BPTT to get a truly conclusive comparison, but I can't think of any reason why it would be better.

Update: I added an answer

Update: here is the loss function I am using. networkOutputs is the number of different classes to choose from.

        for (int n = 0; n < batchSize; n++)
            for (int i = 0; i < networkOutputs; i++)
                cost -= Math.Log(output[n][i]) * targets[n][i];

        cost /= batchSize;

In a recurrent neural network, you would usually forward propagate through several time steps, "unroll" the network, and then back propagate across the sequence of inputs.

Why would you not just update the weights after each individual step in the sequence? (the equivalent of using a truncation length of 1, so there is nothing to unroll) This completely eliminates the vanishing gradient problem, greatly simplifies the algorithm, would probably reduce the chances of getting stuck in local minima, and most importantly seems to work fine. I trained a model this way to generate text and the results seemed comparable to results I have seen from BPTT trained models. I am only confused on this because every tutorial on RNNs I have seen says to use BPTT, almost as if it is required for proper learning, which is not the case.

My only guess for why it is normally propagated through time is that maybe it learns faster or tends to converge on a more accurate solution. I have not yet tried a more traditional RNN with BPTT to get a truly conclusive comparison, but I can't think of any reason why it would be better.

Update: I added an answer

In a recurrent neural network, you would usually forward propagate through several time steps, "unroll" the network, and then back propagate across the sequence of inputs.

Why would you not just update the weights after each individual step in the sequence? (the equivalent of using a truncation length of 1, so there is nothing to unroll) This completely eliminates the vanishing gradient problem, greatly simplifies the algorithm, would probably reduce the chances of getting stuck in local minima, and most importantly seems to work fine. I trained a model this way to generate text and the results seemed comparable to results I have seen from BPTT trained models. I am only confused on this because every tutorial on RNNs I have seen says to use BPTT, almost as if it is required for proper learning, which is not the case.

My only guess for why it is normally propagated through time is that maybe it learns faster or tends to converge on a more accurate solution. I have not yet tried a more traditional RNN with BPTT to get a truly conclusive comparison, but I can't think of any reason why it would be better.

Update: I added an answer

Update: here is the loss function I am using

        for (int n = 0; n < batchSize; n++)
            for (int i = 0; i < networkOutputs; i++)
                cost -= Math.Log(output[n][i]) * targets[n][i];

        cost /= batchSize;