Concept: The entropy is the average information, which can be written mathematically as \(H = {p_1}{\log _2}\frac{1}{{{P_1}}} + {p_2}{\log _2}\frac{1}{{{P_2}}} + \ldots \)
Application: Let the Random variable be ‘L’, i.e.
‘L’ = 1, 2, 3,… is the no. of tosses to get first ‘Head’
Since the dice is fair, the probability of getting a ‘head’ or a ‘tail’ is equal, i.e. ½.
The probability of getting a head at the first toss will be:
\(P\left\{ {L = 1} \right\} = \frac{1}{2}\)
The probability of not getting a ‘head’ at the first but getting it at the second toss will be:
\(P\left\{ {L = 2} \right\} = \frac{1}{2} \times \frac{1}{2} = \frac{1}{4}\)
Similarly, the probability of getting a ‘head’ at the third toss will be:
\(P\left\{ {L = 3} \right\} = \frac{1}{8}\)
\(H = {p_1}{\log _2}\frac{1}{{{P_1}}} + {p_2}{\log _2}\frac{1}{{{P_2}}} + \ldots \)
\(H\left( L \right) = \frac{1}{2}{\log _2}\frac{1}{{1/2}} + \frac{1}{4}{\log _2}\frac{1}{{1/4}} + \frac{1}{8}\log \frac{1}{{1/8}} + \ldots \)
\(= 0 + 1 \cdot \frac{1}{2} + 2 \cdot \frac{1}{4} + 3 \cdot \frac{1}{8} + \ldots \)
The geometric series summation is given by:
\(\frac{a}{{1 - r}} + \frac{{d\; \times \;r}}{{{{\left( {1 - r} \right)}^2}}}\)
With a = 2, d = ½ and r = ½, we get:
\(H\left( L \right) = \frac{2}{{1 - \frac{1}{2}}} + \frac{{\frac{1}{2}\left( 1 \right)}}{{{{\left( {1 - \frac{1}{2}} \right)}^2}}}\)
H(L) = 2
