We have f(x) = {–b2 + (a – 1) b – 2}x + ∫(sin2x + cos4x)dx, a, b ∈ R
f'(x) = {– b2 + (a – 1)b – 2} + (sin2x + cos4x)
Since f(x) is increasing function, so f'(x) ≥ 0
{– b2 + (a – 1)b – 2} + sin2x + cos4x) ≥ 0
{– b2 + (a – 1)b – 2} + 3/4 ≥ 0,
since the minimum
value of sin2x + cos4x is 3/4
{– 4b2 + 4(a – 1)b – 8 + 3} ≥ 0
–4b2 + 4 (a – 1)b – 5 ≥ 0
4b2 – 4(a – 1)b + 5 ≤ 0
So its D < 0
16(a – 1)2 – 80 < 0
(a – 1)2 – 5 < 0
(a – 1)2 < 5
|(a – 1)| < √5
–√5 < (a – 1) < √5
1 – √5 < a < 1 + √5
Hence, the value of a is (1 – √5, 1 + √5).