Let d = OP. Then cos POA = a/d and cos POB = b/d, so POA = acos(a/d) and POB = acos(b/d).
You can then write cos(acos(a/d) + acos(b/d)) = cos(60) = 1/2.
Since cos(x + y) = cos(x) cos(y) - sin(x) sin(y), you can then plug into this formula to solve for d.