In a recent paper on the experimental results of when neural networks learn via directed feedback alignment, https://arxiv.org/pdf/2006.12878.pdf, it was shown that directed feedback alignment achieved results similar, albeit worse, to backpropagation in a variety of tasks. A clear implementation of directed feedback alignment could be found in the origin paper, https://arxiv.org/pdf/1609.01596.pdf. Essentially, it replaces the transpose of the weight matrices used in backpropagation with random constant matrices. Does anyone have any idea why such a method could possibly work?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
This diagram might be useful: (from this paper)
Direct Feedback Alignment (DFA) is a modification of Feedback Alignment (FA), which replaces Backpropogation (BP). With FA, you replace $\mathbf{W_i}$ in the gradient update with a random projection $B_i$. With DFA, you replace $\mathbf{W_i}$ with $B_i$, but you also replace the loss signal $\delta \mathbf{a}_i$ with the global loss signal $\delta \mathbf{a}_y$.
In the paper on FA, they explain why it works at all:
