Entropy is a measure of how predictable the outcome of a random process is, or how much uncertainty one has about an unknown quantity in one's state of knowledge, defined in terms of the probability distribution on possible outcomes or quantities.
Entropy is a property of a probability distribution on the set of possible values of a random process or of an unknown quantity. A probability distribution $P$ quantifies the plausibility of each possible value $x_1,$ $x_2,$ $\dotsc,$ $x_n$ by assigning it a numerical probability $P(x_1),$ $P(x_2),$ $\dotsc,$ $P(x_n)$ between $0$ and $1$; the entropy of $P$ is a single measure loosely summarizing the magnitude of uncertainty about what value the random process might yield, or the unknown quantity might actually be.
The two most common kinds of entropy are:
Shannon entropy, $H(P) := -\sum_i P(x_i) \log_2 P(x_i)$, is the average number of bits per sample that an optimal compression algorithm tuned for the distribution $P$ can compress random samples from $P$ into.
Shannon entropy, named after Claude Shannon for his seminal work on information theory, is most often what unqualified ‘entropy’ refers to outside cryptography in information theory and coding theory. Shannon entropy is useful for estimating the average cost of transmitting samples of $P$ by telegrams that you must pay for by the bit.
Min-entropy, $H_\infty(P) := -\log \max_i \log_2 P(x_i)$, is the number of fair coin tosses that all come up heads with the same probability as the best strategy of guessing a sample from $P$ on the first try.
Min-entropy is the number of fair coin tosses coming up heads that the best guess for a single outcome has the same probability as, and is most often what unqualified ‘entropy’ refers to in cryptography. Min-entropy is useful for estimating an adversary's probability of success at guessing a key in a single trial.
Example. A four-sided die with probability 1/2 of turning up 1, probability 1/4 of turning up 2, and equal probabilities 1/8 of turning up 3 or 4, can be compressed into messages, say for transmission by telegram which costs by the bit, as follows:
- Transmit the face 1 as a
0bit.- Transmit 2 as the bit string
10.- Transmit 3 as
110.- Transmit 4 as
111.The most probable outcome, rolling a 1, has the same probability as a single fair coin toss coming up heads, 1/2. Thus the min-entropy is 1 bit.
The average number of bits per sample in this compression scheme is the sum of each number of bits weighted by its probability. A straightforward calculation shows that this compression scheme is optimal, because the average number of bits coincides with the Shannon entropy, namely 1.75 bits.
This example illustrates a theorem the min-entropy is never greater than the Shannon entropy; that is, Shannon entropy is an upper bound on min-entropy, so it is useful in cryptography as a limit on the best min-entropy you can hope for from a physical system.
Rényi entropy is a generalization that covers both Shannon entropy and min-entropy as instances, but it seldom figures into cryptography. Entropy can be measured in other units such as nats, decibans, etc., if computed with base-e, base-10, etc., logarithms instead of base-2 logarithms, but this is seldom seen in cryptography.
Thermodynamic entropy is related to Shannon entropy. In a thermodynamic system characterized by macroscopic averages such as temperature, pressure, volume, etc., the (thermodynamic) entropy change of the system is defined in terms of the macroscopic heat transfer into or out of it and its temperature change.
A priori, this concept of classical thermodynamics may not be obviously related to information theory or probability distributions, but turns out to have an interpretation in terms of the Shannon entropy of a family of probability distributions.
In the microscopic formulation of statistical mechanics, when a thermodynamic system is described in terms of macroscopic averages, there are many possible microscopic configurations that the system could be in that are compatible with the macroscopic averages. Among the probability distributions on microscopic configurations compatible with the macroscopic averages, the maximum Shannon entropy of any such probability distribution is the (thermodynamic) absolute entropy of the system (with an appropriate choice of logarithm base to make the units commensurate), and a change in entropy coincides with a difference of absolute entropies.
Historically, the term entropy was invented by Rudolf Clausius as a macroscopic property of a thermodynamic system, before Ludwig Boltzmann connected it to microscopic configurations in his H theorem and J. Willard Gibbs expounded on it in the development of statistical mechanics. Claude Shannon later stumbled upon the same formula as Gibbs, but from the perspective of channel coding and information theory. Inspired by Boltzmann and Gibbs, Shannon adopted the name ‘entropy’ and letter H for the property of any probability distribution.
