As the number of spectroscopic stellar surveys start to increase, we aim to adopt new methodologies to automatically infer stellar labels (stellar parameters and chemical abundances) from optical stellar spectra. In this research, we infer four stellar labels using The Payne (Ting, Conroy, Rix, & Cargile, 2019), a neural network that emulates high-resolution synthetic theoretical spectra and enables full-spectrum fitting. We identify and mask spectral pixels that are not modeled accurately due to the limited number of stellar labels and modeling deficiencies in the synthetic spectra. To calibrate the masks for optical spectra, we use the Gaia-ESO benchmark stars. Gaia-ESO is a set of spectra where the four stellar labels we aim to predict are known, namely, the effective temperature, surface gravity, metallicity, and alpha-enhancement. We add masks until we can get our predicted labels to match the true labels, within a given margin of error. With masking, we can achieve accurate and precise stellar parameters for the Gaia-ESO benchmark stars. As a test, we show these masks can also accurately recover stellar parameters from the R-Process Alliance survey.