Shannon Entropy and Information Gain @SerranoAcademy
Shannon Entropy and Information Gain  @SerranoAcademy
Uploaded November 2017 | Updated November 2026, 2 days ago
CORRECTION: at 13:41, the probability is 6.1e-5 and not 4.8e-4 (however, the entropy is 1.75, which is correct). Thank you @dlyChimi!

Learn Shannon entropy and information gain by playing a game consisting in picking colored balls from buckets.

Announcement: New Book by Luis Serrano! Grokking Machine Learning. bit.ly/grokkingML
40% discount code: serranoyt

Accompanying blog post: medium.com/p/5810d35d54b4

0:00 Shannon Entropy and Information Gain
2:22 What ball will we pick?
4:33 Quiz
5:06 Question
5:14 Game
7:17 Probability of Winning
7:45 Products
11:00 What if there are more classes?
12:34 Sequence 2
13:44 Sequence 3
14:57 Naive Approach
15:34 Sequence 1
19:44 General Formula
Shannon Entropy and Information GainHappy 2025, and thank you for your support!What is AdaBoost? Friendly explanation with code!A friendly introduction to Deep Learning and Neural NetworksTraining Latent Dirichlet Allocation: Gibbs Sampling (Part 2 of 2)What is the best approximation to Pi?Why is KL Divergence not symmetric?Live with Gaurav Sen, Josh Starmer, and Luis Serrano!Singular Value Decomposition (SVD) and Image CompressionGenerative model that won the 2024 Physics Nobel Prize - Restricted Boltzmann Machines (RBM)Low Rank Adaptation (LoRA) - How to search for a needle in a much smaller haystackTriple BAM! Live with Jon Krohn, Josh Starmer, and Luis Serrano!
Luis Serrano Academy |

Shannon Entropy and Information Gain

SHARE TO X SHARE TO REDDIT SHARE TO FACEBOOK WALLPAPER