The Evolution of the Number Density of Compact Galaxies