Sorry Alan, that doesn't work. The identity has to be true for all x, not just x = 1/a.
Also, there's no reason why a should equal b.
Start by moving the first term to the rhs, and then take the tangent of both sides.
\(\displaystyle \tan(\tan^{-1}(ax)+\tan^{-1}(bx))=\tan\left(k-\tan^{-1}\left(\frac{1}{x}-\frac{x}{8}\right)\right)\)
(k for the moment, rather than pi/2).
Expand both sides using the tan(A + B) identity. Also, on the rhs, then divide top and bottom by tan(k).
\(\displaystyle \frac{ax+bx}{1-abx^{2}}=\frac{1-(1/x-x/8)/\tan(k)}{1/\tan(k)+(1/x-x/8)}.\)
Now replace k by pi/2 and this becomes
\(\displaystyle \frac{ax+bx}{1-abx^{2}}=\frac{1}{(1/x)-(x/8)}.\)
Cross multiply, multiply out and collect up terms,
\(\displaystyle a+b-\frac{x^{2}}{8}(a+b-8ab)=1.\)
For this to be an identity, true for all values of x, it's necessary that a + b - 8ab = 0, and also that a + b = 1.
Solving those simultaneously leads to
\(\displaystyle a =\frac{2\pm \sqrt{2}}{4}\quad \text{and} \quad b=\frac{2 \mp \sqrt{2}}{4},\)
after which
\(\displaystyle a^{2}+b^{2}=\frac{3}{4}.\)
.