Expand the left hand side
\(abx^2+(a^2+b^2)x+ab=26x^2+[]x + 26\)
So we must have
a*b = 26
a and b are integers, so the possibilities are
a = 1 b = 26
a = 2 b = 13
(or the same with a and b switched).
Hence [] must be either \(1^2+26^2\) or \(2^2+13^2\)
I'll leave you to decide which is smaler.