Small asteroids are lost from the main belt by rotational disruption, catastrophic collisions, chaotic orbital evolution and evolution driven by Yarkovsky radiation forces. However, the timescales of these loss mechanisms are not known. In the inner main belt, the mean sizes of the high-inclination, non-family asteroids increase with increasing orbital inclination (Dermott et al., 2018). Here, we show that this observation is accounted for by the unique resonant structure of the inner main belt that results in the loss of asteroids through the escape hatches at the ν6 secular resonance and the 3:1 Jovian mean motion resonance. From the observed asteroid size and orbital inclination distributions, we show that orbital evolution due to Yarkovsky radiation forces is the dominant loss mechanism for asteroids with diameters 2 < D < 7 km. We also show that Yarkovsky-driven orbital evolution accounts for the observed non-linear size-frequency distributions of the major asteroid families. From the observed asteroid size and orbital inclination correlation, we calculate that, on average over the age of the solar system, the semimajor axes of the asteroids change on a timescale 13.4 Gyr. This timescale is an upper limit because (a) asteroids experience collisional evolution and thus their current sizes are mostly less than their formation sizes and (b) it is possible that the spin directions of the asteroids have experienced reversals with the result that the sense of orbital evolution, either towards or away from the Sun, has not been constant.