Uploaded September 2022 | Updated September 2026, 2 weeks ago
Attention in Natural Language Processing (NLP) is super important mechanism which makes your #AI systems such as chat bots, text predictions, sentimental analysis and more smarter and clever.
Good example is BERT. Attention is all you need! - as one famous article declares.
There are at least 4 different #Attention types in #NLP:
1. Encoder-Decoder attention (applied in Recurrent Neural Network - RNN).
2. Self attention.
3. Bi-Directional attention.
4. Multi-Head attention.
This video mostly explains the theory behind the encoder-decoder attention mechanism works. Here we will cover such aspects as:
- What is Dot-product in this NLP technique.
- Which role is taken by Similarity function for word to vector (wor2vec) approach.
- What is Alignment in attention calculation and how to calculate it.
- How to calculate a Dot-product (two different ways)
- Experiment with Python with word vectors (np.dot vs. np.matmul functions).
Finally, we will dive deeper into Dot-Product Attention with the schema which provides you intuition how this works. We will open this "black box"!
The content of the video:
0:00 - Intro
1:24 - Alignment with Dot-Product
5:04 - Experiments with Word Vectors in Python
12:06 - Dot-Product Attention (mathematical aspect)
In the last part of the video, we will get known about logical steps to calculate a Dot-Product for attention. In mathematical documentation it is called - matrix Z. This is our Attention product!
Here you will find such math operations such as matrix multiplication and transposing matrixes. Also you will find what roles takes embedding, linear, and softmax layers in the attention schema. Additionally, you will understand why we need to pay attention to PAD (padding) tokens as the input to encoder or decoder.
Read more:
1. Attention is all you need: official article in Arxiv: arxiv.org/abs/1706.03762
2. Attention and #Transformers: towardsdatascience.com/all-you-need-to-know-about-attention-and-transformers-in-depth-understanding-part-1-552f0b41d021
3. Attention Masks: lukesalamone.github.io/posts/what-are-attention-masks
Brilliant Course on Udemy, from which I learn all these things I showing you in this video: NLP With Transformers in Python by James Briggs (udemy.com/course/nlp-with-transformers/)
NEXT LESSON: Self-Attention in NLP : youtu.be/pfvYuX0Gjys
#attention #dotproduct #naturallanguageprocessing #encoder #decoder #ai #artificialintelligence #machinelearning #datascience
@DataScienceGarage .
Attention in Natural Language Processing (NLP) is super important mechanism which makes your #AI systems such as chat bots, text predictions, sentimental analysis and more smarter and clever.
Good example is BERT. Attention is all you need! - as one famous article declares.
There are at least 4 different #Attention types in #NLP:
1. Encoder-Decoder attention (applied in Recurrent Neural Network - RNN).
2. Self attention.
3. Bi-Directional attention.
4. Multi-Head attention.
This video mostly explains the theory behind the encoder-decoder attention mechanism works. Here we will cover such aspects as:
- What is Dot-product in this NLP technique.
- Which role is taken by Similarity function for word to vector (wor2vec) approach.
- What is Alignment in attention calculation and how to calculate it.
- How to calculate a Dot-product (two different ways)
- Experiment with Python with word vectors (np.dot vs. np.matmul functions).
Finally, we will dive deeper into Dot-Product Attention with the schema which provides you intuition how this works. We will open this "black box"!
The content of the video:
0:00 - Intro
1:24 - Alignment with Dot-Product
5:04 - Experiments with Word Vectors in Python
12:06 - Dot-Product Attention (mathematical aspect)
In the last part of the video, we will get known about logical steps to calculate a Dot-Product for attention. In mathematical documentation it is called - matrix Z. This is our Attention product!
Here you will find such math operations such as matrix multiplication and transposing matrixes. Also you will find what roles takes embedding, linear, and softmax layers in the attention schema. Additionally, you will understand why we need to pay attention to PAD (padding) tokens as the input to encoder or decoder.
Read more:
1. Attention is all you need: official article in Arxiv: arxiv.org/abs/1706.03762
2. Attention and #Transformers: towardsdatascience.com/all-you-need-to-know-about-attention-and-transformers-in-depth-understanding-part-1-552f0b41d021
3. Attention Masks: lukesalamone.github.io/posts/what-are-attention-masks
Brilliant Course on Udemy, from which I learn all these things I showing you in this video: NLP With Transformers in Python by James Briggs (udemy.com/course/nlp-with-transformers/)
NEXT LESSON: Self-Attention in NLP : youtu.be/pfvYuX0Gjys
#attention #dotproduct #naturallanguageprocessing #encoder #decoder #ai #artificialintelligence #machinelearning #datascience
@DataScienceGarage .
