Radial velocities of ~3000 fundamental mode RR Lyrae stars (RRLs) toward the Southern Galactic bulge are presented, spanning the southern bulge. Distances derived from the pulsation properties of the RRLs are combined with Gaia proper motions to give constraints on the orbital motions of ~1500 RRLs. The majority (~75%) of the bulge RRLs have orbits consistent with these stars being permanently bound to <0.5 kpc from the Galactic Center, similar to the bar. However, unlike the bulge giants, the RRLs exhibit slower rotation and a higher velocity dispersion. The higher velocity dispersion arises almost exclusively from halo interlopers passing through the inner Galaxy. Once the 25% of the sample represented by halo interlopers is cleaned, we can discern two populations of bulge RRLs in the inner Galaxy. One population of RRLs is not as tightly bound to the Galaxy (but is still confined to the inner ~3.5 kpc), and is both spatially and kinematically consistent with the barred bulge. The second population is more centrally concentrated and does not trace out the bar. We discuss interpretations for these two seemingly different populations of bulge stars.