Art of the Problem | Claude Shannon's Information Entropy (Physical Analogy) @ArtOfTheProblem | Uploaded 10 years ago | Updated 9 hours ago
Entropy is a measure of the uncertainty in a random variable (message source). Claude Shannon defines the "bit" as the unit of entropy (which is the uncertainty of a fair coin flip). In this video information entropy is introduced intuitively using bounce machines & yes/no questions.

Note: This analogy applies to higher order approximations, we simply create a machine for each state and average over all machines!