If there are n people, you would count the number of handshakes as nC2.
nC2= \(\frac{n(n-1)}{2\cdot1}\)
If there were 10 people, there would be (10*9)/2=45 handshakes. Thats not enough, so lets try a bigger number. 12 people: (12*11)/2=66 handshakes so there were 12 people.