Well, I am no stats expert but... this is a binomial distribution.
Each time the die is rolled there is a 1/6 = 0.16 repeater chance of rolling a 5.
You are asking if 23 fives is a probable outcome.
I am going to jump over to a binomial calculator
http://stattrek.com/online-calculator/binomial.aspx
I put in the info and it spits out some cumulative probabilities. (Maybe you use tables for this?)
Anway it tells me that $$p(x\ge 23)=0.0605$$
This means that if you throw a fair die 100 times and count the number of 5s
and you do this a great many times, then you will get 23 or more fives approx 6% of the time.
SO, at a five percent significance level this is not significant.