Earthquake_Simulation_Workflow_Engine_on_NeSIs_New_HPC.pdf (1.93 MB)
Earthquake Simulation Workflow Engine on NeSI’s New HPC
presentation
posted on 2019-05-15, 00:19 authored by Jonney HuangCybershake NZ is a project that simulates possible ruptures for all known faults in New Zealand
based on the national earthquake rupture forecast that currently consists of a total of ~12,000
simulations. This prompted us the need for fully automated workflow management to maximise the
throughput and utilization of the NeSI allocation.
Each ground motion simulation comprises multiple steps that need to be executed with specific
dependencies. To automate the workflow of ~12,000 simulations, we need tools that can
periodically monitor the job status and handle complex logic for job submission with a good wall
clock time estimation, dependency checks and automated testing.
Our workflow engine is based on variety of technologies such as Python, Bash, SQLite, etc. and
has enabled us to run the entire Cybershake NZ within 3 weeks, which could have been over 3
months.
In this talk, we present the method and technologies used for the workflow engine and the
challenges encountered during the development.
