Question

For a discrete memory less source containing K symbols, the upper bound for the entropy is
1. 1/K
2. log₂K
3. log₁₀ K
4. log_{2 ­}(1-K)

Answer 1

Correct Answer - Option 2 : log₂K

Entropy is the average information content per symbol in a group of symbols.

For M symbols with probability P₁, P₂,…P_i, the entropy (H) is given by:

\(H = - \mathop \sum \nolimits_{i = 1}^M {P_i}{\log _2}\left( {{P_i}} \right)\;bits/symbol\) 

For a discrete memoryless source containing K symbols with equal probability of occurrence,

maximum entropy is there and is given by:

H_max = log₂ K bits/symbol

Note:

Range of Entropy is 0 ≤ H ≤ log₂ M

If the symbol rate is r_s symbol/sec, the source information rate is:

R = r_s H bit/sec