Earthquake Simulation Workflow Engine on NeSI’s New HPC
2019-05-15T00:19:59Z (GMT) by
Cybershake NZ is a project that simulates possible ruptures for all known faults in New Zealand based on the national earthquake rupture forecast that currently consists of a total of ~12,000 simulations. This prompted us the need for fully automated workflow management to maximise the throughput and utilization of the NeSI allocation.
Each ground motion simulation comprises multiple steps that need to be executed with specific dependencies. To automate the workflow of ~12,000 simulations, we need tools that can periodically monitor the job status and handle complex logic for job submission with a good wall clock time estimation, dependency checks and automated testing.
Our workflow engine is based on variety of technologies such as Python, Bash, SQLite, etc. and has enabled us to run the entire Cybershake NZ within 3 weeks, which could have been over 3 months.
In this talk, we present the method and technologies used for the workflow engine and the challenges encountered during the development.