Problem that is easy.

X = 3-sqrt(7) ~~ 0.354249    and    Y = 3+sqrt(7) ~~ 5.64575

X = 3+sqrt(7) ~~ 5.64575     and    Y = 3-sqrt(7) ~~ 0.354249

x^2y^2 + x^2 + y^2 + 2xy = 40  →  x^2y^2 + (x + y)^2  = 40     (1)

xy + x + y = 8   →  y( x + 1)  = 8 - x   →  y  = [ 8 - x] / [ x + 1]      (2)

Put (2)  into (1)

x^2  [ 8 - x]^2 / [x + 1]^2  + [ x + [8 - x] / [ x + 1] ] ^2 = 40     simplify

x^2 [ x^2 - 16x + 64] / [ x + 1]^2  + [ x (x + 1)/(x + 1) + [ 8 - x] / [x + 1] ]^2  = 40

x^2[ x^2 - 16x + 64] + [ x^2  + 8 ]^2  =  40[x + 1]^2

x^4 - 16x^3 + 64x^2  + [ x^2 + 8]^2 = 40[x + 1]^2

x^4 - 16x^3 + 64x^2 + x^4 + 16x^2 + 64  = 40[x^2 + 2x + 1]

2 x^4-16 x^3+80 x^2+64 = 40 x^2 + 80 x + 40

2x^4 - 16x^3 + 40x^2  -80x +24  = 0

x^4 - 8x^3 + 20x^2 - 40x + 12 = 0

This has no rational solutions

The real solutions are  x = 3 ±√7, y  = 3 -/+ √7

The non-real solutions are x = 1 ± i*√5  , y= 1 -/+ i*√5