Uploaded October 2022 | Updated September 2026, 2 weeks ago
This video clearly explains how Self-Attention works in NLP (Natural Language Processing). The main idea is calculate relationship between an actual token (word) and previous tokens in the same text sequence.
Self-attention is broadly used in areas such as text summarization and text generation. Self-attention was proposed by researchers at Google Research and Google Brain. It was proposed due to challenges faced by encoder-decoder in dealing with long sequences.
The self-attention model allows inputs to interact with each other (i.e calculate attention of all other inputs). To better understand self-attention mechanism, I suggest to check for material about Transformers in NLP, because: the Transformer model revolutionized the implementation of attention by dispensing of recurrence and convolutions and, alternatively, relying solely on a self-attention mechanism.
You can read more about it here: machinelearningmastery.com/the-transformer-attention-mechanism
As always in attention calculation (as we demonstrated in Decoder-Encoder attention in the last video lesson), we need to calculate a Dot product, which consists of K (Keys) transposed and Q (Queries).
The content of the video:
0:00 - Intro to attention and other mechanisms
0:48 - Principal scheme explanation of Self-Attention
6:40 - Quick example
Learn more in this video right now and enjoy!
I hope the attention intuition introduced in this video let you understand how self-attention mechanism works in more details. If yes, I would be so glad of that.
The first lesson: Encoder-Decoder Attention in NLP : youtu.be/z5upsjfVU9c
#attention #nlp #transformers #selfattention #dotproduct #ai #datascience #datasciencegarage #softmax
@DataScienceGarage
This video clearly explains how Self-Attention works in NLP (Natural Language Processing). The main idea is calculate relationship between an actual token (word) and previous tokens in the same text sequence.
Self-attention is broadly used in areas such as text summarization and text generation. Self-attention was proposed by researchers at Google Research and Google Brain. It was proposed due to challenges faced by encoder-decoder in dealing with long sequences.
The self-attention model allows inputs to interact with each other (i.e calculate attention of all other inputs). To better understand self-attention mechanism, I suggest to check for material about Transformers in NLP, because: the Transformer model revolutionized the implementation of attention by dispensing of recurrence and convolutions and, alternatively, relying solely on a self-attention mechanism.
You can read more about it here: machinelearningmastery.com/the-transformer-attention-mechanism
As always in attention calculation (as we demonstrated in Decoder-Encoder attention in the last video lesson), we need to calculate a Dot product, which consists of K (Keys) transposed and Q (Queries).
The content of the video:
0:00 - Intro to attention and other mechanisms
0:48 - Principal scheme explanation of Self-Attention
6:40 - Quick example
Learn more in this video right now and enjoy!
I hope the attention intuition introduced in this video let you understand how self-attention mechanism works in more details. If yes, I would be so glad of that.
The first lesson: Encoder-Decoder Attention in NLP : youtu.be/z5upsjfVU9c
#attention #nlp #transformers #selfattention #dotproduct #ai #datascience #datasciencegarage #softmax
@DataScienceGarage
