It's auxiliary equation is
m2 + 2m - 1 = 0
⇒ m = \(\frac{-2\pm\sqrt{4+4}}2=-1\pm\sqrt2\)
C.F. = e-x (C1cos 2x + C2sin 2x)
P.I. = \(\frac1{D^2+2D-1}\)x2 e2x cos x
\(=\frac{1}{(D+1)^2-2}\) x2 e2x cos x
= e2x \(\frac{1}{(D+3)^2-2}\) x2 cos x
= x2e2x \(\frac1{D^2+6D+7}\) cos x + 2x e2x {\(\frac d{dD}\frac1{D^2+6D+7}\)} cos x + e2x {\(\frac {d^2}{dD^2}\frac1{D^2+6D+7}\)} cos x
= x2e2x \(\frac1{D^2+6D+7}\) cos x + 2xe2x (\(\frac{-(2D+6)}{(D^2+6D+7)^2}\)) cos x + e2x(\(\frac{2(2D+6)^2}{(D^2+6D+7)^3}\) cos x - \(\frac{2}{(D^2+6D+7)^2}\)) cos x
= \(\frac{x^2e^{2x}}6\frac{D-1}{D^2-1}\)cos x + 2xe2x - \(\frac{-(2D+6)}{36(D+1)^2}cos x\) + \(\frac{8e^{2x}}{216}\frac{(D+3)^2}{(D+1)^3}cos x\) - \(\frac{2e^{2x}}{36}\frac{1}{(D+1)^2}cosx\)
= \(-\frac1{12}x^2e^{2x}\)(D cos x - cos x) - \(\frac19\)xe2x\(\frac{(D+3)(D-1)^2}{(D^2-1)^2}\)cos x + 27 e2x \(\frac{(D+3)^2(D-1)^3}{(D^2-1)^3}\) cos x - \(\frac{e^{2x}}{18}\frac{(D-1)^2}{(D^2-1)^2}\) cos x
= \(\frac1{12}\)x2e2x(sin x + cos x) - \(\frac{xe^{2x}}{36}\)(D3 + D2 + -5D + 3) cos x - \(\frac{27}8\)e2x(D5 + 3D4 - 6D3 - 10D2 + 21D - 9) cos x
- \(\frac{e^{2x}}{72}\)(D2 - 2D + 1) cos x
= \(\frac1{12}\)x2e2x(sin x + cos x) - \(\frac{xe^{2x}}{36}\)(6 sin x + 2 cos x) - \(\frac{27}8\)e2x(-28 sin x + 4 cos x) - \(\frac{e^{2x}}{72}\)x 2 sin x
= \(\frac1{12}\) x2e2x (sin x + cos x) - \(\frac{xe^{2x}}{18}\) (3 sin x + cos x) - \(\frac{27}2\)e2x(-7sin x + cos x) - \(\frac{e^{2x}}{36}\) sin x
complete solution is y = C.F. + P.I.
⇒ y = e-x(C1 cos 2x + C2 sin 2x) + \(\frac1{12}\)x2e2x(sin x + cos x) - \(\frac{xe^{2x}}{18}\)(3 sin x + cos x) - \(\frac{27}2\)e2x(-7 sin x + cos x) - \(\frac{-e^{2x}}{36}\) sin x