(7/9a+9/7b) 2 - ab

Looks like you got it right:

 

Expand the following:
((7 a)/(9)+(9 b)/(7))^2-a b

((7 a)/(9)+(9 b)/(7)) ((7 a)/(9)+(9 b)/(7)) = ((7 a)/9) ((7 a)/9) + ((7 a)/9) ((9 b)/7) + ((9 b)/7) ((7 a)/9) + ((9 b)/7) ((9 b)/7) = (49 a^2)/(81)+a b+a b+(81 b^2)/(49) = (49 a^2)/(81)+2 a b+(81 b^2)/(49):
(49 a^2)/(81)+2 a b+(81 b^2)/(49)-a b

Grouping like terms, (49 a^2)/(81)+2 a b+(81 b^2)/(49)-a b = (81 b^2)/(49)+(2 a b-a b)+(49 a^2)/(81):
(81 b^2)/(49)+(2 a b-a b)+(49 a^2)/(81)

2 (a b)-a b = a b:
Answer: |(81 b^2)/(49)+ab+(49 a^2)/(81)