Surfaces of the asteroids (25143) Itokawa, (162173) Ryugu, and (101955) Bennu, the targets of recent sample return missions, are dominated by boulder fields without the presence of fine-grained ponded deposit, indicating an active regolith removal process at work. Here, based on recent laboratory and space experimental results, we show that, at 1 au heliocentric distance, asteroids smaller than 1 kilometer in radius experience a net loss of surface fine-grained material. This is because the regolith loss driven by electrostatic dust lofting dominates the production from fragmentation caused by thermal fatigue and meteoroid impacts, mainly because of the low-gravity environment of these small bodies. Sub-kilometer Main Belt Asteroids likely become regolith-free Near Earth Asteroids as they migrate inwards. The electrostatic regolith removal is expected to be coupled to the orbital and rotational evolution of small bodies driven by thermal radiation processes. In conjunction with rotation, space weathering, and other processes, electrostatic regolith size-sorting and removal affects the surface properties of small asteroids, contributing to their spectral diversity.