This is a hard problem , there are only some approximations
like \(\frac{π}{12}\times \left ( \frac{{R}_{2}^2}{{r}_{1}^2} \right ) - \frac{π}{12} \times \left ( \frac{{R}_{2}}{{r}_{1}} \right )\)
R2 is the large circle , r1 is the small circle
you can use an approximiate calculator
http://www.engineeringtoolbox.com/smaller-circles-in-larger-circle-d_1849.html
You may find this interesting
http://www.jstor.org/pss/2688509
http://www2.stetson.edu/~efriedma/cirincir/
http://en.wikipedia.org/wiki/Circle_packing_in_a_circle