1. Open the compasses to a distance greater than half the distance between F and G.
2. Put the compass point on F and draw a semicircle that crosses line FG.
3. Repeat 2. with the compass point on G.
4. The two semicircles will intersect at two points, so use the straight edge to draw a line between them. This line will be perpendicular to line FG.
The calculator doesn’t understand your implicit multiplication, so you need to enter a*b=11, not ab =11. If you do this the calculator produces the correct two (complex) results.
In your case, the calculator simply ignored the ab=11 part and assigned an arbitrary variable, r1, to b and then solved for a as a = 6 - b → 6 - r1.