3x2 + 4x + 12 = 0
Use the quadratic formula to find the roots.
r = [ -4 + sqrt( 42 - 4·3·12 ) / ( 2·3 ) = [ -4 + sqrt( -128 ) / ( 6 ) = [ -4 + 8 · i · sqrt(2) ] / 6
s = [ -4 - sqrt( 42 - 4·3·12 ) / ( 2·3 ) = [ -4 - sqrt( -128 ) / ( 6 ) = [ -4 - 8 · i · sqrt(2) ] / 6
r2 = ( [ -4 + 8 · i · sqrt(2) ] / 6 )2 = [ -112 - 64·i·sqrt(2) ] / 36 (which can be reduced)
s2 = ( [ -4 - 8 · i · sqrt(2) ] / 6 )2 = ...
r2 + s2 = ...
You CAN use the quadratic formula, but that takes a lot of time. Instead, you can use Vieta's Formulas. Vieta's for quadratics states that for a quadratic \(ax^2 + bx + c\), the sum of the roots is \(-\frac{b}{a} \) and the product of the roots is \(\frac{c}{a}\). In case you want to find out more, here's an AoPS wiki link: https://artofproblemsolving.com/wiki/index.php/Vieta%27s_Formulas
Anyway, \(r^2 + s^2 = (r+s)^2 - 2rs\), which means that we can use Vieta's. Thus, \(r^2+s^2 = \left( -\frac{4}{3} \right)^2 - 2\cdot \frac{12}{3} = \frac{16}{9} - 8 = \boxed{-\frac{56}{9}}\). Remember, when the problem is about some expression with the roots, if you can't factor the polynomial, try Vieta's.