Can someone help me understand how to get the answer for this question 5xy-3x^2+6y And please do it step by step and elaborate, thanks :)

Melody is correct....there's not much you can do here, unless you would like to set this = 0 and graph it...in that case we get a "rotated conic"  (a hyperbola) which is kinda' cool.....(the presence of the "xy" term implies a "rotated" graph)

Are you sure this is the whole question Emily?  I don't think that there is anything that you can do with that.

Maybe the answer is "Can't be simplified"  I slip in trick questions like that all the time with my students.