We can mutiply out (a-bi)2 to get a2 - 2abi - b2.
Since a and b are positive integers, the -10i on the right side of the equation must come from -2abi.
Therefore -10i = -2abi so ab = 5.
And now we see this problem is literally impossible because 5 can only factor into 1 and 5 or 5 and 1, and (1 - 5i)2 is -24 - 10i while (5 - i)2 is 24 - 10i. So unless I'm dumb and missed something, which wouldn't suprise me, this question is either has a typo or is impossible.
Assuming it's a typo and the left side is actually adds to 24 - 10i and not 21 - 10i, the answer is 5 - i.