Inferring the solar photospheric magnetic field from Zeeman polarization data involves many steps and assumptions, each with varying degree of impact on the accuracy of the result. It has been long known that the treatment of unresolved structures and instrumental scattered light will influence the inferred strength and direction of the field. The impact of chosen assumptions for the HMI Pipeline data reduction is most visibly manifest as a sign-change in the (local) horizontal field direction in plage areas according to East/West hemisphere location, as presented in Pevtsov+2021. The ramifications for science are most apparent when considering large-scale magnetic structures from synoptic-derived vector data products. The challenge to mitigation is, of course, that we do not know the answer — and “hare & hound” approaches using synthetic data require more than just a sunspot model, they must include the subtle radiative transfer and instrumental effects that are at play here. In this poster, metrics to calculate the magnitude of these issues fairly directly from the inversion output are presented, based on time-series analysis of presumably steady solar features. The approach is demonstrated for SDO/HMI and Hinode/SOT-SP, but applicable to other instruments; the impacts are quantified for both weak- and strong-flux areas. We present some avenues being considered for removing or at least lessening the impact of these issues, with the goal of achieving improved time-series analysis and synoptic vector-field maps. This work is carried out with support from NASA grants 80NSSC19K0317, 80NSSC18K0180, Solar B FPP Phase E, the U. Michigan SOLSTICE DRIVE Center, and NASA Contract NAS5-02139 (HMI) to Stanford University.