The top and bottom vertices of the square, (as seen in the diagram), remain in contact with the top and bottom sides of the rectangle respectively, so the orientation of the square will be the same as it slides from one end of the rectangle to the other.
Slide the square to the left so that the left hand vertex of the square comes into contact with the left hand side of the rectangle.
Two triangles are formed and their interiors will be will be areas of the rectangle that can't be covered by the square.
The two triangles are congruent. (Call one of the angles of one of the triangles theta, and work out, in terms of theta, the other three angles of the two triangles. That will show that they are similar. They have the same hypotenuse so they are congruent.)
Now slide the triangle to the right so that the right hand vertex of the square comes into contact with the right hand side of the rectangle. Again two triangles are formed and they will congruent with the two triangles on the left.
The total area of the rectangle that can't be covered by the square will be the combined area of the four triangles, which will be four times the area of a single triangle because of the congruency. Notice that the second dimension of the rectangle, the 18, is irrelevant,(so long as it's big enough for the rectangle to contain the square).
The hypotenuse of the triangle(s) is 8, let the lengths of the other two sides be x and y, then
\(x^{2}+y^{2}=64, \\ x + y = 10.\)
Solving simultaneously,
\(x = 5 \pm \sqrt{7}, \quad y = 5 \mp\sqrt{7},\)
so the area of a single triangle will be
\((5+\sqrt{7})(5-\sqrt{7})/2=(25-7)/2=9,\)
meaning that the total area that can't be covered by the square is 36.