Fuzzy Dark Matter (FDM), consisting of ultralight bosons (mb ~ 10-22 eV), is an intriguing alternative to Cold Dark Matter. Numerical simulations that solve the Schrodinger-Poisson (SP) equation show that FDM halos consist of a central solitonic core, which is the ground state of the SP equation, surrounded by an envelope of interfering excited states. These excited states also interfere with the soliton, causing it to oscillate and execute a confined random walk with respect to the halo center of mass. Using high-resolution numerical simulations of a 6.6 X 109 Msun FDM halo with mb = 8 x 10-23 eV in isolation, we demonstrate that the wobbling, oscillating soliton gravitationally perturbs nuclear objects, such as supermassive black holes or dense star clusters, causing them to diffuse outwards. In particular, we show that, on average, objects with mass < 0.3 % of the soliton mass (Msol) are expelled from the soliton in ~ 3 Gyr, after which they continue their outward diffusion due to gravitational interactions with the soliton and the halo granules. More massive objects (> 1 % Msol), while executing a random walk, remain largely confined to the soliton due to dynamical friction. We also present an effective treatment of the diffusion, based on kinetic theory, that accurately reproduces the outward motion of low mass objects and briefly discuss how the observed displacements of star clusters and active galactic nuclei from the centers of their host galaxies can be used to constrain FDM.