Creating a living archive of Aotearoa New Zealand’s climate model data

Over the last year, approximately 0.5 petabytes of climate model data has been generated using the New Zealand Earth System Model, or ‘NZESM’ [Behrens et al.]. Needless to say, this amount of data cannot be analysed easily without postprocessing. It also requires archiving and for this we have the new NeSI ‘nearline’ archiving system.

Climate model data are the product of a concerted effort between dozens of centres around the world and the data is extremely valuable to determine the impact of climate change on society. Due to the amount of data, postprocessing is numerically expensive, which justifies the need for archiving results. This is equally true for other climate modelling work done at NIWA, for example as part of the next Intergovernmental Panel on Climate Change (IPCC) report; part of the larger ‘coupled model intercomparison project’, CMIP6.

The NZESM itself is made up of dynamic ocean, sea-ice and atmospheric ‘general circulation models’ and each of these requires their own methods or postprocessing, each of which will be briefly presented and demonstrated. I will briefly discuss the model itself before illustrating how the main results can be processed and presented in point-and-click format before the data is archived. I will also briefly illustrate how the nearline system is used. Together these represent the ‘living archive’ referred to in the presentation title; one that can be examined easily using already-generated output.

The individual simulation themselves can take several months to run and so it is essential that we are able to monitor the progress of the model in real time. I will also present results from a successful NeSI consultancy project where the run speed of this monitoring software (‘afterburner’) was sped up by a factor of almost 40 [Pletzer, 2020] using the Slurm workload manager and Cylc [Oliver et al., Pletzer] on NeSI’s Māui platform.

References

● Erik Behrens et al., Local Grid Refinement in New Zealand's Earth System Model: Tasman Sea Ocean Circulation Improvements and Super‐Gyre Circulation Implications, Journal of Advances in Modeling Earth Systems, https://doi.org/10.1029/2019MS001996, 2020.

1. H. Oliver et al., “Workflow Automation for Cycling Systems: The Cylc Workflow Engine”, Computing in Science & Engineering Vol 21, Issue 4, https://doi.org/10.1109/MCSE.2019.2906593, July/Aug 2019.

2. Pletzer, ‘rosesnip’ software, https://github.com/pletzer/rosesnip, 2020.

Acknowledgements

1. The UK Met Office-led Unified Model Consortium.

2. NeSI scientific and technical staff.

3. The Deep South National Science Challenge for funding.

Jonny has a PhD in computational physics from the University of Bath and has worked in UK academia and government as a climate scientist as well as in private industry as an environmental consultant. He plays classical viola and is a very slow runner.

Alexander Pletzer is High Performance Computing Research Software Engineer for NeSI at NIWA. Alex plays ping-pong and windsurfs whenever he’s not running jobs on NeSI.