HELP!!

Are you allowed to use a calculator?

AB = 7*sqrt(6).

I do not think the earlier answer is correct.

Here is a graphical solution;

It shows and answer of approximately 2.04 units

P is a point outside of equilateral triangle ABC. PA=3, PB=4, PC=5. Find AB. 

cos rule:

$\begin{array}{|rcll|} \hline 4^2 &=& 3^2+x^2-2*3*x\cos(y) \\ 16 &=& 9+x^2-6x\cos(y) \\ 6x\cos(y) &=& -16 + 9+x^2 \\ 6x\cos(y) &=& x^2-7 \\ \mathbf{\cos(y)} &=& \mathbf{\dfrac{x^2-7}{6x}} \\ \hline \end{array} \begin{array}{|rcll|} \hline \sin^2(y) &=& 1-\cos^2(y) \\ \sin^2(y) &=& 1-\dfrac{(x^2-7)^2}{36x^2} \\ \sin^2(y) &=& \dfrac{36x^2-(x^2-7)^2}{36x^2} \\ \sin(y) &=& \dfrac{ \sqrt{36x^2-(x^2-7)^2} } {6x} \\ \sin(y) &=& \dfrac{ \sqrt{36x^2-x^4+14x^2-49} } {6x} \\ \mathbf{\sin(y)} &=& \mathbf{\dfrac{1}{6x} \sqrt{50x^2-x^4-49 } } \\ \hline \end{array}$

cos rule:

$\begin{array}{|rcll|} \hline 5^2&=& 3^2+x^2-2*3x\cos(60^\circ+y) \\ 25&=& 9+x^2-6x\cos(60^\circ+y) \\ 6x\cos(60^\circ+y) &=& x^2-16 \\ \mathbf{\cos(60^\circ+y)} &=& \mathbf{\dfrac{x^2-16} {6x}} \\\\ \cos(60^\circ)\cos(y)-\sin(60^\circ)\sin(y) &=& \dfrac{x^2-16} {6x} \\ \dfrac{1}{2}\cos(y)-\dfrac{\sqrt{3}}{2}\sin(y) &=& \dfrac{x^2-16} {6x} \quad | \quad \times 2 \\ \cos(y)- \sqrt{3}\sin(y) &=& \dfrac{2x^2-32} {6x} \\ \dfrac{x^2-7}{6x}- \sqrt{3}\dfrac{1}{6x} \sqrt{50x^2-x^4-49 } &=& \dfrac{2x^2-32} {6x} \quad | \quad \times 6x \\ x^2-7 - \sqrt{3} \sqrt{50x^2-x^4-49 } &=& 2x^2-32 \\ -\sqrt{3}\sqrt{50x^2-x^4-49 } &=& x^2-25 \qquad \text{square both sides} \\ 3(50x^2-x^4-49) &=& (x^2-25)^2 \\ 150x^2-3x^4-147 &=& x^4-50x^2+625 \\ \ldots \\ 4x^4-200x^2+772 &=& 0 \quad | \quad : 4 \\ \mathbf{x^4-50x^2+193} &=& \mathbf{0} \\\\ x^2 &=& \dfrac{50\pm \sqrt{50^2-4*193}} {2} \\ x^2 &=& \dfrac{50\pm \sqrt{4*432}} {2} \\ x^2 &=& \dfrac{50\pm 2\sqrt{432}} {2} \\ x^2 &=& 25 \pm \sqrt{432} \\ x &=& \sqrt{ 25 \pm \sqrt{432} } \\ x &=& \sqrt{ 25 - \sqrt{432} } \qquad \text{point outside the triangle} \\ x &=& \sqrt{ 4.21539030917} \\ \mathbf{x} &=& \mathbf{2.05314157066} \\ \hline \end{array}$

$\mathbf{AB\text{ is } \approx 2.05 }$