We explore the galaxy structure of four tidal disruption event (TDE) host galaxies on 30 pc to kpc scales using HST WFC3 multi-band imaging. The star formation histories of these hosts are diverse, including one post-starburst galaxy (ASASSN-14li), two hosts with recent weak starbursts (ASASSN-14ae and iPTF15af), and one early type (PTF09ge). Compared to early type galaxies of similar stellar masses, the TDE hosts have higher central surface brightnesses and stellar mass surface densities on 30-100 pc scales. The TDE hosts do not show the large, kpc-scale tidal disruptions seen in some post-starburst galaxies; the hosts have low morphological asymmetries similar to those of early type galaxies. The lack of strong asymmetries are inconsistent with a recent major (~1:1 mass) merger, although minor (≲1:3) mergers are possible. Given the time elapsed since the end of the starbursts in the three post-burst TDE hosts and the constraints on the merger mass ratios, it is unlikely that a bound supermassive black hole binary (SMBHB) has had time to coalesce. The TDE hosts have low central (<140 pc) ellipticities compared to early type galaxies. The low central ellipticities disfavor a strong radial anisotropy as the cause for the enhanced TDE rate, although we cannot rule out eccentric disks at the scale of the black hole gravitational radius of influence (~1 pc). These observations suggest that the high central stellar densities are a more important driver than SMBHBs or radial anisotropies in increasing the TDE rate in galaxies with recent starbursts.