See the following image :
ED = r
Angle HEF = 120° so angle DEF = 60°
And angle EDF = 90°
And angle EFD = 30°
So triangle EDF is a 30 - 60 -90 right triangle with ED = r, DF = √3 r and EF = 2r
The area of the smaller equilateral triangle = 3(1/2)(ED)^2* sin (120°) = 3(1/2)(r)^2(√3 / 2)
The area of the larger equilateral triangle = 3(1/2) (EF)^2 * sin (120°) = 3(1/2) (2r)^2 (√3 / 2)
So we have that
(3√3)/4 * [ (2r)^2 - r^2] = 36
(3√3)/4 * [ 4r^2 - r^2 ] = 36 multiply through by 4
3√3 [ 3r^2] = 144
9√3 r^2 = 144
r^2 = 144 / (9√3)
r^2 = 16/ √3
r = 4 / 4√3 cm ≈ 3.04 cm