I am not very good at trig but I will try to solve it.
\((cos\space x-5sin\space x)^2=Acos\space 2x+Bsin\space 2x+C\)
\(cos^2\space x - 10sin\space x\space cos\space x + 25sin^2\space x= A(cos^2\space x - sin^2\space x)+B(2 sin\space x\space cos\space x)+C\)
The form of left hand side doesn't fit in the form of right hand side......