A determination of the dynamical evolution of the asteroid belt is difficult because the asteroid belt has evolved since the time of asteroid formation through mechanisms that include: (1) catastrophic collisions, (2) rotational disruption, (3) chaotic orbital evolution and (4) orbital evolution driven by Yarkovsky radiation forces. The timescales of these loss mechanisms are uncertain and there is a need for more observational constraints. In the inner main belt, the mean size of the non-family asteroids increases with increasing inclination. Here, we use that observation to show that all inner main belt asteroids originate from either the known families or from ghost families, that is, old families with dispersed orbital elements. We estimate that the average age of the asteroids in the ghost families is a factor of 0.34 ± 0.02 less than the Yarkovsky orbital evolution timescale. However, this orbital evolution timescale is a long-term average that must allow for the collisional evolution of the asteroids and for stochastic changes in their spin directions. By applying these constraints on the orbital evolution timescales to the evolution of the size-frequency distribution of the Vesta asteroid family, we estimate that the age of this family is greater than 1.3 ± 0.1 Gyr and could be comparable with the age of the solar system. By estimating the number of ghost families, we calculate that the number of asteroids that are the root sources of the meteorites and the near-Earth asteroids that originate from the inner main belt is about 20.