__Concept__**:**

The cutoff frequency of TE_{mn} mode in the rectangular waveguide is given as:

\({f_c} = \frac{C}{{2\pi }}\sqrt {{{\left( {\frac{{m\pi }}{a}} \right)}^2} + {{\left( {\frac{{n\pi }}{b}} \right)}^2}} \)

f_{c} for TE_{10} will be:

\({f_{c\left( {10} \right)}} = \frac{C}{{2\pi }}\sqrt {{{\left( {\frac{{1\pi }}{a}} \right)}^2}} = \frac{C}{{2a}}\) ---(1)

f_{c} for TE_{20} will be:

\({f_{c\left( {20} \right)}} = \frac{C}{{2\pi }}\sqrt {{{\left( {\frac{{2\pi }}{a}} \right)}^2}} = \frac{C}{a}\) ---(2)

f_{c} for TE11 will be:

\({f_{c\left( {11} \right)}} = \frac{c}{{2\pi }}\sqrt {{{\left( {\frac{\pi }{a}} \right)}^2} + {{\left( {\frac{\pi }{n}} \right)}^2}} \)

\( = \frac{c}{2}\sqrt {{{\left( {\frac{1}{a}} \right)}^2} + {{\left( {\frac{1}{b}} \right)}^2}} \;\) ---(3)

__Calculation__**:**

Given:

\({f_c}\left( {T{E_{11}}} \right) = \frac{{{f_c}\left( {T{E_{10}}} \right) + {f_c}\left( {T{E_{20}}} \right)}}{2}\)

From equation 1, 2, 3, we can write:

\(\frac{c}{2}\sqrt {{{\left( {\frac{1}{a}} \right)}^2} + {{\left( {\frac{1}{b}} \right)}^2}} = \frac{{\frac{c}{{2a}} + \frac{c}{a}}}{2}\)

\(\sqrt {{{\left( {\frac{1}{a}} \right)}^2} + {{\left( {\frac{1}{b}} \right)}^2}} = \frac{1}{{2a}} + \frac{1}{a} = \frac{3}{{2a}}\)

\({\left( {\frac{1}{a}} \right)^2} + {\left( {\frac{1}{b}} \right)^2} = \frac{9}{{4{a^2}}}\)

a = √5

\({\left( {\frac{1}{b}} \right)^2} = \frac{5}{{4{a^2}}}\)

\(b = \frac{{2a}}{{\sqrt 5 }}\)

b = 2 cm