Earthquake Simulation Workflow Engine on NeSI’s New HPC

Cybershake NZ is a project that simulates possible ruptures for all known faults in New Zealand based on the national earthquake rupture forecast that currently consists of a total of ~12,000 simulations. This prompted us the need for fully automated workflow management to maximise the throughput and utilization of the NeSI allocation. <div><br></div><div>Each ground motion simulation comprises multiple steps that need to be executed with specific dependencies. To automate the workflow of ~12,000 simulations, we need tools that can periodically monitor the job status and handle complex logic for job submission with a good wall clock time estimation, dependency checks and automated testing. </div><div><br></div><div>Our workflow engine is based on variety of technologies such as Python, Bash, SQLite, etc. and has enabled us to run the entire Cybershake NZ within 3 weeks, which could have been over 3 months. </div><div><br></div><div>In this talk, we present the method and technologies used for the workflow engine and the challenges encountered during the development.</div>