The relationship between outflows and galaxy properties, particularly at high redshifts, is an important piece of the galaxy evolution puzzle. We use a sample of 27 GRBs at redshift z = 2-6 to probe the outflows in their respective host galaxies (mass range M*/Msun ~ 109-11) and search for possible relations between the outflow properties and those of the host galaxies such as M*, SFR, and specific SFR. First, we consider the following three outflow properties — outflow column density (Nout), maximum outflow velocity (Vmax), and normalized maximum velocity (Vnorm = Vmax/Vcirc,halo, where Vcirc,halo is the halo circular velocity). We observe clear trends of Nout and Vmax with increasing SFR in high-ion-traced outflows, with a stronger (> 3-sigma) Vmax-SFR correlation. Next, we find that the estimated mass outflow rate and momentum flux of the high-ion outflows scale with SFR and can be supported by the momentum imparted by star formation (i.e. supernovae and stellar winds). The kinematic correlations of warm outflows (traced by high-ions) with SFR are similar to those observed for star-forming galaxies at low redshifts. The correlations with SFR are weaker in low-ion species. This, along with the lower detection fraction in low-ions (irrespective of the SFR), indicates that the outflow is primarily high-ion dominated. We also observe a strong (> 3-sigma) trend of normalized velocity (Vnorm) decreasing with halo mass and increasing with sSFR, suggesting that outflows from low-mass halos and high-sSFR galaxies are most likely to escape and enrich the outer CGM and IGM with metals. These observations have begun to fill the gap in our understanding of outflows from low-mass galaxies at high redshifts and provide a stepping stone for future studies with the ELT/GMT and JWST.