Question

Consider a discrete memory less source with alphabet S = s₀, s₁, s₂, s₃, s₄, ...  and respective probabilities of occurrence P = {1/2, 1/4, 1/8, 1/16, 1/32}. The entropy of the source (in bits) is _______.

Answer 1

Concept: The entropy of the source is the average information. 

Calculation: The entropy of the source is:

 \(\rm H(s)=-s_0 log_2⁡\ s_0-s_1 log_2⁡\ s_1-s_2 log_2⁡\ s_2 …………\)

\( \Rightarrow {\rm{H}}\left( {\rm{s}} \right) = \frac{1}{2} × 1 + \frac{1}{4} × 2 + \frac{1}{8} × 3 + \frac{1}{{16}} × 4. \ldots \ldots \ldots .. \)

\(\Rightarrow {\rm{H}}\left( {\rm{s}} \right) = \frac{1}{2}\left[ {1 +2 × {\rm{\;}}\frac{1}{2} + 3 × \frac{1}{ 2^2 } + 4 × \frac{1}{{{2^3}}} \ldots \ldots \ldots \ldots .} \right] \)

Now, the expansion of

\({\left( {1 - {\rm{x}}} \right)^{ - 2}} = 1 + 2{\rm{x}} + 3{{\rm{x}}^2} + 4{{\rm{x}}^3}{\rm{\;}} \ldots \ldots \ldots .\)

Comparing with

\(1 + 2 × \frac{1}{2} + 3 × \frac{1}{{{2^2}}} + 4 × \frac{1}{{{2^3}}}\)

We see, \({\rm{x}} = \frac{1}{2}\)

Thus, 

\(\begin{array}{l} = {\left( {1 - \frac{1}{2}} \right)^{ - 2}}\\ = {\left( {\frac{1}{2}} \right)^{ - 2}} = {2^2} = 4 \end{array}\)

Now,

H(s) = 0.5 × 4 = 2 bits/symbol