Transformers are the state-of-the-art model for seq2seq tasks.

image.png

image.png

One Basic method for self-attention

  1. money bank grows
  2. river bank flows

image.png

image.png

Similarity: dot product

image.png

image.png

Next: Normalise the weights

image.png