Shannon entropy: mathematically quantifying how little you know.

Shannon’s entropy formula measures the average uncertainty, surprise, or information contained in a set of possible outcomes. It assigns more entropy to systems where outcomes are harder to predict and less to those dominated by a few likely possibilities. A perfectly predictable source has zero entropy; a balanced, unpredictable one carries more information per observation. The formula gave information theory a quantitative foundation and made it possible to analyze how efficiently data can be represented, transmitted, and compressed. It is used primarily in data compression, communications engineering, and machine learning.