With the advent of space-based observatories, we are facing a big data problem in astronomy. Machine learning serves as an efficient approach to discover patterns from such data. Supervised learning techniques (e.g. neural networks) often surpass unsupervised approaches for performing classification tasks on complex data. However, labeling large datasets is an onerous and time-consuming process that is often prohibitively expensive. In this study, we show that a deep neural network trained on crudely labeled astronomical data can be leveraged to improve the quality of data labeling in a time efficient manner that minimizes human intervention. We use SoHO/MDI magnetic evolution videos, approximately labeled for emergence/non-emergence. We train a convolutional neural network (CNN) to perform the classification task and only manually verify the labels of videos, which are incorrectly classified by the model. We iterate this process until there is no change in classification accuracy. After performing a full manual verification, we find that the large majority of videos where the model succeeded were indeed properly labeled. We also show that apart from performing the classification task, the model is able to identify when emergence occurs. Our results demonstrate that big datasets do not need to be perfectly labeled initially for supervised learning. Instead, focusing only on failed examples can refine the labeling. This subset is by definition smaller than the full set and thus requires less manual work. Solar magnetic flux-emergence is often associated with space weather events that can potentially have a disruptive impact on long-distance communications. The complex interaction of solar magnetic elements often limits the ability of conventional image-processing techniques to identify flux emergence. Our CNN’s ability to identify both the emergence event and its starting time hints at the possibility of using deep learning to enable flux emergence prediction.