My square patio is tiled with square tiles, all the same size. All the tiles are gray, except the tiles along the two diagonals, which are all yellow. (The corners are yellow, the center is yellow, and all the tiles along the diagonal in between are yellow.) If there are $20$ yellow tiles, how many gray tiles are there?