I think this is supposed to be :
2x^2-72xy+23y^2-80x-60y-125=0
We need to first calculate this:
cot2Θ = [A - C] / B = [2 - 23]/[-72] = [-21]/[-72] = [7/24]
And, from trig, we have that cos2Θ = 7/25
And using trig again, we can caclulate sinΘ and cosΘ, thusly
sinΘ = √[(1 - cos2Θ)/2 ] = √[(1 - 7/25)/2] = √[9/25] = 3/5
cosΘ = √[(1 + cos2Θ)/2 ] = √[(1 + 7/25)/2] = √[16/25] = 4/5
Now....we need to make the following substitutions
x = cosΘ X - sinΘ Y and y = sinΘ X + cosΘ Y .... or, just .......
x = (4/5)X - (3/5)Y and y = (3/5)X + (4/5)Y
So we have
2[(4/5)X - (3/5)Y]^2-72[ (4/5)X - (3/5)Y][(3/5)X + (4/5)Y]+23[(3/5)X + (4/5)Y]^2-80[ (4/5)X - (3/5)Y]-60[ (3/5)X + (4/5)Y -125=0-80[ (4/5)X - (3/5)Y]
And term by term, we have.......
[(32/25)X^2 - (48/25)XY + (18/25)Y^2] +
[(-864/25)X^2 - (504/25)XY + (864/25)Y^2] +
[(207/25)X^2 +(552/25)XY + (368/25)Y^2] +
[48Y - 64X] +
[-36X - 48Y ] -
125 = 0
Simplifying the above, we get
50Y^2 - 25X^2 -100X - 125 = 0 divide through by 25
2Y^2 - x^2 - 4x - 5 = 0 complete the square on x
2Y^2 - [x^2 + 4x + 4 - 4] = 5
2Y^2 - (x +2)^2 = 1
And we have the recognizable equation of a hyperbola.
Here's a picture of both the original graph with the xy term and the "normal" graph without rotation...
https://www.desmos.com/calculator/rrkgvzswml
BTW...the original graph's rotation can be found thusly:
cot2Θ = [A - C] / B = [2 - 23]/[-72] = [-21]/[-72] = [7/24]
cot-1(7/24) =73.739795291688°
So half of this is the angle of rotation ≈ 36.87°