Building a standardized Observing System Simulation Experiment (OSSE) framework for Mars

assess the beneﬁt of future atmospheric observations and establish the roadmap


Executive Summary
We advocate that the Decadal Survey recommends the NASA Science Mission Directorate to develop a rigorous Observing System Simulation Experiment (OSSE) framework for Mars, to optimize future atmospheric observations.Atmospheric conditions on Mars are a potential hazard source for landing missions.Errors in the estimates of atmospheric density profiles, inadequate knowledge of wind vertical structure and dust concentration as a function of height are likely causes of uncertainty at the landing site on the order of kilometers.An operational real-time weather forecasting capability for Mars would reduce such uncertainties, carrying enormous benefits to future robotic missions, and would be an invaluable prerequisite for human missions.
A real-time forecasting capability relies upon three fundamental components: a critical mass of observing systems, a data assimilation system (DAS), and a global forecast model.The DAS allows the model to ingest the data effectively, optimizing the observational information content, and transforming them into a gridded representation of the atmosphere at a given time, called an 'analysis'.The analysis is the best estimate of the atmospheric state for that time, and also represents a set of 'initial conditions' from which a global model can be initialized, to predict a future state of the atmosphere.The connection between analysis and forecast represents the foundation of modern weather forecasting.
However, from the point of view of a forecast system, not all observations are equally impactful, partially because of the problem of "observational error correlation", one important research topic in data assimilation development.For the Earth, partly due to the spontaneous and deregulated development of observations and forecast capabilities worldwide for more than half a century, the use of observations in contemporary operational forecast systems is suboptimal, with many potentially useful data being underutilized.On the contrary, Mars atmospheric scientists are in the unique situation of designing the next-generation observing systems by learning from the experience gathered on the Earth, so as to assure that the future instruments are specifically optimized to give the maximum benefit to a future weather forecast capability.
An immensely powerful tool that has been firmly established by atmospheric scientists on the Earth is represented by a properly designed OSSE framework.A realistic OSSE framework can not only quantify the benefit of future data types, be them surface based or space borne, but can also help design and optimize an entire observational network.Furthermore, OSSEs can provide deep insights into an atmosphere's behavior, by addressing conceptual problems of its intrinsic predictability and delineating the regions or features of the atmosphere which are more sensitive to additional data and would benefit from a denser sampling.The difficulties posed by OSSEs are fundamentally different for Earth and Mars.For Earth, the enormous data volume imposes a tremendous constraint on any innovation in the observing systems: it is very hard for a single sensor to impact the skill.For Mars, the problem is the opposite: almost any additional instrument will exert some impact.However, OSSEs can help to evaluate the cost/benefit for every sensor and suggest optimal data configuration and density.
The purpose of this white paper is to provide an introduction to a rigorously designed OSSE framework, explain the underlying problems and challenges, and engage the Mars community to collaborate with Earth Atmospheric scientists in order to develop a joint-OSSE framework for Mars with the largest consensual basis possible.An OSSE infrastructure would increase the understanding of the Martian atmosphere, would help NASA to optimize instrument specifications and orbit choice, providing the maximium benefit for a given expenditure of resources, and could even help establishing a roadmap for a future real-time weather forecasting capability.

Introduction
The foundation of a global weather forecast capability for the Earth is rooted within the inseparable and deep connection between a Data Assimilation System (DAS) and a global forecast model.A global atmospheric DAS is designed to produce a gridded representation of the atmosphere, called "analysis" (e.g., Daley 1992).The analysis is based on existing data, is constrained by physical laws, and represents the best estimate of the unknown state of the atmosphere.For the Earth, a global analysis is represented by atmospheric data on a regularly-spaced tridimensional grid, in which the horizontal resolution is generally on the order of a tenth of a degree, corresponding to about ten kilometers, while the vertical resolution is determined by about a hundred unevenly spaced vertical levels up to 100 km of height, generally encompassing the whole mesosphere.Such high resolution is made possible by the huge amount of available data: for any assimilation cycle (i.e., every few hours), almost 10 7 observations are assimilated in a DAS, comprising both satellite and conventional observations.However, even these millions of assimilated data are a very small subset of the total volume of observation.At any given time, the analysis represents the state of the atmosphere that can be used as 'initial conditions' for a forecast model.There are two types of analyses: 'operational analyses' and 'reanalyses'.The former are produced routinely by operational weather forecast centers, and are subjected to frequent changes and updates in the DAS and the model, making comparisons across time problematic.In contrast, the latter is an analysis performed for a very long time (years or decades) with a frozen system, and therefore is suitable to produce long term statistics and comparisons between events several years apart.
A global forecast model numerically solves a set of equations representing the fundamental conservation laws (energy, momentum, mass, constituents with changes of state, etc.) as a function of time.The ability of a forecast system to predict future states of the atmosphere can be rigorously quantified a posteriori with a set of metrics to assess the distance between the predicted and observed state of the atmosphere as a function of time.For the Earth, more than half a century of numerical weather predictions has provided several important lessons that could be of immense benefit for the future of Mars exploration.We know that the quality of the forecast is determined by a number of intimately connected factors which go far beyond the 'goodness' of a model and the quality of the observations.Less intuititive concepts such as data distribution, data representativeness, and observational error correlation play a prominent, even if less obvious, role.
In order to establish a weather forecasting capability for Mars it is essential to drastically increase the observational basis.However, it is very likely that a lack of coordination between instrument developers and limited resources from NASA or other agencies could lead to less than optimal results.A properly built Observing System Simulation Experiment framework could allow each instrument team to test their own instrument concept multiple times so as to evaluate its impact in a comparable and standardized way.This would enable the construction of the observational network in the most efficient way possible.This paper is divided as follow: the next section provides a review of OSEs and OSSEs and describes some factors hindering an optimal use of observations, section 3 illustrates a possibile pathway to design an OSSE framework for Mars, and section 4 states the conclusion of this work.

OSEs and OSSEs
OSSEs represent the expansion of the so-called Observing System Experiments (OSEs), also called 'data impact studies', that are used to evaluate the impact of existing sensors on the Earth.An OSE framework requires a credible weather forecast capability, i.e., the most comprehensive set of observing systems, a high-quality global model, and a DAS.The DAS can assimilate the millions of observations produced operationally, generating the initial conditions for the forecast model.If one instrument is removed from or added to the assimilation phase, the two sets of initial conditions ("with" or "without" that instrument) will slightly differ.As a consequence of the initial difference, even if small, forecasts initialized from the two sets will rapidly diverge as a function of forecasting time.The comparison of two forecasts (with and without the instrument under scrutiny) allows a rigorous quantification of the instrument's impact.This procedure is routinely used for Earth in order to evaluate the existing sensors' impacts and optimize a weather forecasting capability (e.g., Reale et al. 2018).One problem that OSEs have revealed is the counter-intuitive paradox that quality of analysis and forecast can be negatively affected by data, because of the problem of observation error correlation in data assimilation.Additional data are beneficial only in regions with strong gradients and elevated rates of change, while they can reduce the quality the analysis in meteorologically inactive areas.Despite the fact that only few percents of satellite observations are assimilated, some observations, even if intrinsically excellent, can degrade the forecast quality.The problem, often arising when oversampling meteorologically stagnant regions of the atmosphere, is well understood theoretically, but it is extroardinarily difficult to solve in practice for the Earth, due to the very large number of existing observing systems.For Mars, an OSSE framework would be beneficial in designing future instruments so as to minimize the occurrence of these problems from the beginning.
An OSSE framework is conceptually similar to an OSE, but designed to assess the impact of instruments that do not yet exist, and is substantially more complex.OSSEs have been used for more than 30 years as a tool to evaluate the benefit of future instruments for the Earth (Atlas et al. 1985;Arnold and Dey 1986).However, compared to earlier studies, contemporary OSSEs demand a very rigorous setup, to avoid the risk of unrealistic assessments.For the Earth, a realistic OSSE framework is computationally expensive and requires a long preparation work, but once established it can serve the community for a very long time.For example, a previous effort to standardize OSSEs in 2005 resulted from an inter-Agency and international collaboration involving NASA, NOAA and the European Center for Medium-Range Weather Forecasts (ECMWF), was called Join-OSSEs (Masutani et al. 2007), and was utilized for ten years: in fact, works based on that configuration were still being produced in the mid 2010s (e.g., Nolan et al 2013;Kleist and Ide 2015).Around 2017, atmospheric scientists at NASA GSFC have created a new OSSE framework for the Earth, which is destined to serve the community for many years to come (Gelaro et al. 2017).
The term OSSE has been used quite liberally to indicate any kind of work relating to future instruments, but experiments carried without a rigorous framework carry modest or no benefits.Two essential requirements for a modern OSSE framework include standardization and durability, with the former allowing sensor developers to use the same framework for different instruments, so that results are comparable, while the latter ensuring the usability of the same framework multiple times for the same instrument during different development phases.
Therefore, to build a rigorous OSSE framework is a lengthy and demanding task that necessitates, at a minimum, the following tools: a) Two different global atmospheric forecast models, b) A Data Assimilation System, joined with one of the two global models and representing the core of the global forecast system; c) A global reanalysis of the atmosphere; d) A long simulation of the atmosphere produced by the other global model, to be used as the so-called "Nature Run" (NR); e) A procedure to extract synthetic observations and their error estimates from the NR.
An OSSE framework requires two completely different atmospheric models, one to be used as NR, to avoid the so-called 'twin problem', an excessively optimistic estimate of future instruments' benefit when the NR and the forecast model used with the DAS are the same (Atlas 1997).A Nature Run needs to be a free-running high-quality, high-resolution simulation of the atmosphere in a free-running mode, without assimilation.The model needs to be run for a relatively long time, with only external (e.g., solar) radiative forcings and physical constraints, so that the memory of the specific initial conditions is forgotten.What differentiates a NR from a climatological run is that the NR must be produced at the highest possible resolution possible, in that it must represents a sequence of meaningful instantaneous states that need to be as close as possible to the 'weather' states.A two-year long simultation at a given resolution would be more suitable to serve as a NR than a simulation five times longer, but at half horizontal resolution.To be useful, the NR must not just have reasonable mean climatological values, but its high-frequency internal variability must compare well with reanalyses and in-situ observations.
From the NR, synthetic observations can be extracted, with their error estimates.These artificial observations must mimic the statistical properties of true observations and, once generated, can be assimilated in the DAS to produce an analysis from which forecasts can be initialized.The validation of these forecasts, however, is performed against states of the Nature Run, instead of states of the atmosphere.

A pathway towards an OSSE framework for Mars
The procedure to establish an OSSE framework comprises the following conceptual steps: 1) Evaluation/Selection of the DAS, forecast model, and reanalysis; 2) Evaluation/selection of the Nature Run (to be performed against reanalyses); 3) Extraction of synthetic observations for existing instruments; 4) Assimilation of data from simulated existing instruments; 5) Calibration; 6) Extraction of synthetic observations for future instruments; 7) Assimilation of data from simulated future instruments; 8) Evaluation of forecast skill (with/without) against states of the Nature Run.
These steps are not sequential, but intensively recursive, so that many iterations are necessary to achieve a stable configuration that can satisfy the requirements of standardization and durability.Figure 1 provides a simplified visual representation of an OSSE framework.
Step 1 involves an accurate evaluation and choice among candidate global models and data assimilation systems, produced with various metrics, to design the forecast system.A data assimilation system based on an analysis correction scheme was used by Montabone et al. (2013) to produce the first reanalysis for Mars, encompassing 3 Martian years.A more recent DAS based on Ensemble Kalman Filter was used to develop the Ensemble Mars Atmosphere Reanalysis System (EMARS), described by Greybush et al. (2019).EMARS was developed by a vast multiinstitutional collaboration and provides a comprehensive estimate of the Martian atmosphere by incorporating data provided by two instruments orbiting Mars, the Thermal Emission Spectrometer (TES) which operated on the Mars Global Surveyor (MGS) from 1999 to 2004, and the Mars Climate Sounder (MCS) operating on the Mars Reconnaissance Orbiter (MRO) since 2006.While there is no overlap between TES and MCS observations, the datasets are consistent and have shown very good agreement.The Atmospheric Model associated with EMARS is the model now known as NASA Ames Mars Global Climate Model, derived from the former Geophysical Fluid Dynamics Laboratory Mars Model (Wilson et al. 2002;Zalucha et al 2010), which had been extensively used with a DAS (Hoffman et al. 2010;Greybush at al. 2012).
Among forecast models, this team is examining a version of the 3-Dimensional Laboratoire de  Meteorology (e.g., Madeleine et al. 2011).The Mars LMD GCM (also referred to as LMD-M) is part of a modular set of planetary atmosphere models (with which shares a finite-difference dynamical core), and has been used for simulations of ancient Mars (Wordsworth et all 2012) and current Mars (e.g., Madeleine et al. 2012;Navarro et al 2014).As such, the LMD is a highly portable and well tested modeling framework.However, it needs to be coupled with a DAS to become part of the forecast system within a Mars OSSE framework.From an OSSE perspective it would be desirable if all Mars GCM developers considered coupling their models with a DAS.
Step 2, the selection and evaluation of the NR, is extraordinarily important, and needs to be achieved on two fronts: a comparison of statistical properties of the NR against reanalyses, and a phenomenological approach involving actual atmospheric features and instantaneous states.For the Earth, the previous NR adopted in the Joint OSSE framework was extensively evaluated from a phenomenological point of view with focus on the representation of tropical cyclones (Reale et al. 2007).Almost eight years after the production of the ECMWF NR, NASA decided to create a new NR.The new NR was comprehensively evaluated by a team of 20 scientists at Goddard (Gelaro et al. 2015) and the phenomenological evaluation with focus on tropical cyclones was performed by Reale et al. (2017).For Mars, a possible NR candidate could be the finite-volume cubed-sphere NASA AMES Mars GCM (Bertrand et al. 2020) particularly suitable for its treatment of dust, and the scalability on high-end computer resources which has allowed horizontal resolutions up to c1440 (solving 1440 waves, corresponding to approximately 0.125 o × 0.125 o at the Equator), but the evaluation has not started yet.However, it would be desirable that several models are investigated in depth.The NR evaluation, as previously done for the Earth NRs, should be conducted from two points of view: statistical properties of the Martian climate and a phenomenological approach, focused on specific events, such as Martian baroclinic transient eddies on different scales, global dust storms, etc.
Steps 3, 4 and 5 are necessary to make sure that the skill obtained by assimilating synthetic data from simulated existing sensors is comparable to the skill obtained by assimilating real data from the corresponding actual sensors.Calibration is essential to ensure that the results, which will be obtained by assimilating data from simulated future sensors, are realistic.Steps 6, 7 and 8 represent the core of the OSSE methodology.They can assess the potential of a future sensor to improve existing weather forecasting skill, either for a category of events (e.g., hurricanes for Earth, dust storms for Mars) and/or globally.
As noted in Figure 1, an established OSSE framework should be envisioned as a multi-component system whose purpose is to provide answers to specific questions, and quantified solutions to different problems.For Mars, the specific questions that could be asked from a properly designed OSSE framework are: • What is the skill that could be ideally achieved with a very large amount of observations?
• What are the intrinsic predictability limits of the Martian atmosphere?
• Are some regions of the Martian atmosphere more predictable than others?
• What are the limits to predictability imposed by different seasons?
• What types of instruments would benefit the forecast skill more than others?
• What instrument configurations could be more beneficial?
• What is the minimum amount of instruments that would allow a prescribed skill level?
In summary, the first outcome of a collaborative OSSE effort for Mars should be the selection of the specific forecast system and NR, resulting from a team assessment.Once established, it would become possible to assimilate existing data into the designated forecast system, and assess current predictability levels.Then it would be possible to start true OSSE experiments, exploring impacts of sensors (infrared, microwave) and data types (e.g., temperature, wind profiles, dust opacity, surface pressure), and establishing potential limits of predictability.

Conclusions
The purpose of this white paper is to raise the attention on the potential benefits of a rigorous OSSE framework to improve our knowledge of the Martian atmosphere.The understanding of the OSSE procedure for the Earth is profound, and it would be beneficial that the lessons learned are shared, in the form of a broad collaboration between Mars Atmospheric Scientists and Earth Scientists who have a documented experience in OSSEs.The benefits would affect the development of current and future missions to Mars and allow NASA to obtain the maximum return of investment from future atmospheric observing systems.The ultimate goal of an OSSE framework should be to establish the critical threshold of observing systems that would enable real-time weather forecasting for Mars.However, there would be a number of intermediate returns, most important the confidence that the potential of future observing systems would be exploited to the maximum level, intrinsically and within next-generation reanalysis projects, such as the successors of EMARS.