We know that the integers modulo 13 form a field, so we can operate on systems almost like usual. Equalities will just mean congruence modulo 13 .
From the first equation we obtain ca=−2ab−bc and therefore the second equation gives
ab+2bc−2ab−bc=2b(−2ab−bc)
that is
b(a−c)=2b2(2a+c)
The third equation gives
ab+bc−4ab−2bc=8b(−2ab−bc)
that is:
b(3a+c)=8b^2(2a+c)
We know that b≠0 (because b is a positive integer less than 13 ), so we can simplify it off. If 2a+c=0 , then also a−c=0 , but this implies a=c=0 , which is excluded. Thus we obtain
2b=(a−c) / (2a+c)
8b=(3a+c) / (2a+c)
Thus we need
4(a−c) / (2a+c)=(c+3a) / (2a+c)
and therefore (4a−4c)=(c+3a) , that is, a=5c .
Now we can substitute in 2ab+bc+ca=0 to get
10bc+bc+5c^2=0
Since c≠0 we obtain 11b+5c=0 , so 2b=5c Multiplying by 7 yields 14b=35c and so b=9c
Now we can substitute in
ab+2bc+ca=2abc
to get:
45c^2+18c^2+5c^2=90c^3
Thus:
3c^2=12c^3
and so −c=3 , that is, c=10 . Hence a=5c=11 and b=9c=12
Solution:
a=11,b=12,c=10
and so a+b+c≡7(mod13)