I tried setting the side length to a. Then, I used the length given and got an equation for the third small triangle. Then I got confused on how to solve that. Is there any other way?
Call the side of the square, S
So...the area of the square = S^2
The area of the small triangle at the bottom left = S* [ sqrt (4^2 - S^2) ] /2 / 2 (1)
The area of the small triangle at the top right = S* [ sqrt (5^2 - S^2)] /2 (2)
And the area of the small triangle at the top left = [ S - sqrt (4^2 - S^2)] [ S - sqrt (5^ - S^2) ] / 2 (3)
And the area of the right triangle = 6 (4)
So
(1) + (2) + (3) + (4) = S^2
S* [ sqrt (16 - S^2 ] /2 + S* [ sqrt (25 - S^2)] / 2 + [ S - sqrt (16 - S^2)] [ S - sqrt (25 - S^2) ] / 2 + 6 = S^2
S* [ sqrt (16 - S^2 ] + S* [ sqrt (25 - S^2)] + [ S - sqrt (16 - S^2)] [ S - sqrt (25 - S^2) ] + 12 = 2 S^2
S* [ sqrt (16 - S^2 ] + S* [ sqrt (25 - S^2)] + [ S - sqrt (16 - S^2)] [ S - sqrt (25 - S^2) ] + 12 = 2 S^2
S* [ sqrt ( 16 - S^2) ] + S*[ sqrt (25 - S^2) ] + S^2 - S*[ sqrt (16 - S^2)] - S* [ sqrt (25 - S^2)] +
sqrt [ ( 16 - S^2) (25 - S^2) ] + 12 = 2S^2
sqrt [ (16 - S^2) (25 - S^2)] = S^2 - 12 square both sides
(16 - S^2) ( 25 - S^2) = S^4 - 24S^2 + 144
400 - 41S^2 + S^4 = S^4 - 24S^2 + 144
17S^2 - 256 = 0
17S^2 = 256
S^2 = 256 / 17