Thank you Alan, I shall study your solution,
Here is my solution:
Solve:
\(y''-2y'+y=(x^2+1)e^x\)
Big Hint:
The solution will be of the from
\(y=(Ax^4+Bx^3+Cx^2)e^x \\~\\ \text{Where A,B and C are constants.}\)
\( y=(Ax^4+Bx^3+Cx^2)e^x\\~\\ y'=[Ax^4+(4A+B)x^3+(3B+C)x^2+2Cx]e^x\\~\\ y''=[Ax^4+(4A+4A+B)x^3+(12A+3B+3B+C)x^2+(6B+2C+2C)x+2C]e^x\\ y''=[Ax^4+(8A+B)x^3+(12A+6B+C)x^2+(6B+4C)x+2C]e^x\\~\\ y''-2y'+y\\ \begin{array}{rrrrrrr} =e^x[&Ax^4&+&(8A+B)x^3&+&(12A+6B+C)x^2&+&(6B+4C)x&+&2C&\\ &+(-2A)x^4&+&(-8A-2B)x^3&+&(-6B-2C)x^2&+&(-4C)x&\\ &+Ax^4&+&Bx^3&+&Cx^2&&&&&]\\~\\ =e^x[&0x^4&+&0x^3&+&12Ax^2&+&6Bx&+&2C&]\\~\\ \end{array}\\ so\\ y''-2y'+y=e^x[12Ax^2+6Bx+2C]\\~\\ \)
\([12Ax^2+6Bx+2C]e^x=(x^2+1)e^x\\~\\ 12A=1 \qquad 6B=0 \qquad 2C=1\\ A=\frac{1 }{12}\qquad \;\; B=0 \qquad \;\; C=\frac{1}{2}\\~\\ so\;\;if\;\;\\ y''-2y'+y=(x^2+1)e^x\\ then\\ y=\left(\dfrac{x^4}{12}+\dfrac{x^2}{2}\right)e^x \)
.