Correct Answer - `-sqrt(2) le lambda le sqrt(2), alpha = n pi, npi + (pi)/(4)`
Given, `lambda x + ("sin" alpha) y + ("cos" alpha) z =0`
`x +("cos" alpha)y + ("sin" alpha)z = 0`
`"and "-x +("sin"alpha) y-("cos" alpha)z =0` has non-trivial solution.
`therefore Delta =0`
`rArr lambda (-"cos"^(2)alpha - "sin"^(2)alpha) -"sin"alpha(-"cos"alpha +"sin"alpha)+ "cos" alpha("sin" alpha + "cos" alpha)=0`
`rArr -lambda + "sin" alpha"cos"alpha +"sin"alpha "cos"alpha-"sin"^(2)alpha + "cos"^(2) alpha =0`
`rArr lambda = "cos" 2alpha + "sin"2alpha [ because -sqrt(a^(2) +b^(2)) le a "sin" theta + b"cos" theta le sqrt(a^(2) + b^(2))]`
`therefore -sqrt(2) le lambda le sqrt(2) " "...(i)`
Again, when `lambda = 1, "cos"2alpha + "sin" 2alpha =1`
`rArr (1)/(sqrt(2)) "cos" 2alpha + (1)/(sqrt(2)) "sin" 2alpha = (1)/(sqrt(2))`
`rArr "cos" (2alpha -pi//4) = "cos" pi//4`
`therefore 2alpha -pi//4 = 2pi +- pi//4`
`rArr 2alpha = 2n pi - pi//4 + pi//4 "or" 2alpha =2n pi + pi//4 + pi//4`
`therefore alpha = n pi "or" n pi + pi//4`