21.2 Relative, joint, and conditional entropy, and mutual information 437
21.2 Relative, joint, and conditional entropy, and mutual information
The properties of VN entropy (henceforth to be referred to simply as “entropy”) are
listed in this section, along with those of related entropy variants that represent the
quantum counterparts of similar definitions in classical theory.
To recall the basics, the previous section showed that the VN entropy is nonneg-
ative (S(ρ) ≥ 0), is zero for systems in pure states (S(ρ) = S(|ψ ψ|) = 0), and has
a maximum of log n for systems of dimension n made of a uniform superposition of
states (S(ρ) = log n). Finally, the entropy is invariant under basis transformation, i.e.,
S(TρT
+
) = S(ρ), where T is a unitary matrix, which illustrates that quantum informa-
tion is an incompressible feature in quantum systems.
I shall now describe more properties of VN entropy.
Relative entropy
Our starting point in the investigation is the definition of relative entropy. This notion is
the quantum equivalent to the classical relative entropy, or Kullback–Leibler distance,
described in Chapter 5. Assume two quantum systems with density operators ρ and σ .
The relative entropy between these two systems, noted S(ρσ )isgivenby:
S(ρσ ) = tr(ρ log ρ) −tr(ρ log σ ). (21.19)
For any operator, the kernel is the space described by the set of eigenvectors having zero
for eigenvalues, and the support is the space described by the set of eigenvectors with
nonzero eigenvalues (the union of kernel and support being the complete eigenvector
space). In the definition in Eq. (21.17), the term tr(ρ log σ ) is finite only in the case
where the support of ρ does not intersect with the kernel of σ, or is fully included in the
support of σ (meaning (ρ log σ )
ii
≥ 0 for all diagonal matrix elements). In the contrary
case, we have −tr(ρ log σ ) =+∞, and the relative entropy is infinite. As a key property
of relative entropy, we have
S(ρσ ) ≥ 0, (21.20)
where the equality stands uniquely for the case ρ = σ . Such a property is referred to as
Klein’s inequality. The basic proof of Klein’s inequality is provided in Appendix U. The
consequence is that for two quantum systems with density operators ρ and σ ,wehave
S(ρ) = tr(ρ log ρ) ≥ tr(ρ log σ ), a property that we shall use later when considering
additivity rules for entropy.
Composite system in pure state
Consider, next, a composite system made of two subsystems A, B in states |ψ
A
, |ψ
B
,
and assume that it is in a pure joint state, namely |ψ=|ψ
A
|ψ
B
.Callρ
A
,ρ
B
the
density operators of the subsystems A, B. Then we have the property
S(ρ
A
) = S(ρ
B
). (21.21)