2
$\begingroup$

Learning the weights of logistic regression using gradient descent is quite intuitive. The input $x$ is multiplied with the weight $w$ to produce $y$, and we know the true target $\hat{y}$. Therefore, during backpropagation, we tweak the value of $w$ so that the next $y$ is closer to $\hat{y}$.

A self-attention module has a query, key, and value matrices trained with the same target. How are these respective weights learned during gradient descent?

Improve this question
$\endgroup$
6
  • $\begingroup$ It works exactly the same way, those keys, queries, etc are also values that you multiply, add, etc. $\endgroup$
    – Tim
    Commented Nov 29, 2022 at 14:47
  • $\begingroup$ Okay, but then how are the query, key, and value weights unique? What's stopping them from all having the same value? $\endgroup$
    – desert_ranger
    Commented Nov 29, 2022 at 17:59
  • 3
    $\begingroup$ By initializing them randomly. But if that is not clear, you should probably learn more about gradient descent and backpropagation in neural networks, because it's the same for all neural networks. $\endgroup$
    – Tim
    Commented Nov 29, 2022 at 18:48
  • $\begingroup$ I apologize, I wasn't clear above. My question was basically, how are the three different weights learned without a specific target? For instance, how does a query weight become formed without a target query? I think this has something to do with the transformer architecture and not gradient descent. $\endgroup$
    – desert_ranger
    Commented Nov 29, 2022 at 19:37
  • $\begingroup$ It works exactly like learning any other parameter for any other neural network. It would probably be easier if you start with a simple fully-connected network. You can try Andrew Ng's lecture youtube.com/watch?v=mO7BpWmzT78 $\endgroup$
    – Tim
    Commented Nov 29, 2022 at 20:21

1 Answer 1

Reset to default
2
$\begingroup$

As Tim said in a comment, it works exactly the same way as the other parameters are learned using backprop.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.