Machine learning bears the potential to revolutionize observational cosmology, especially with the big data anticipated of upcoming surveys that aim to scrape the dregs of the uncertainty-dominated sky. However, the validity of analyses utilizing such techniques hinges on first solving a number of open and open-ended methodological questions. I present a multifaceted case study of the rewards and risks of machine learning in the context of galaxy survey cosmology with broad-band photometric redshifts by reviewing the results of Schmidt, Malz, & Soo, et al. (2020) and Malz & Hogg (2021). I identify prerequisites to the usage of probabilistic data products of machine learning to cosmological problems and outline priorities for future directions of data-driven astronomical research, highlighting in particular a need for methods yielding likelihoods of data conditioned on parameters rather than posteriors of parameters conditioned on data.