Embracing a changing eResearch Landscape: NeSI’s horizons beyond traditional HPC

In 2024 NeSI was fully focused on provisioning its new HPC infrastructure. This marks the third major infrastructure provisioning process NeSI has embarked on. This presentation focuses on two of the key approaches taken and shares the experience and learnings from the process.

The first approach is around provisioning of the HPC and its core services using the Infrastructure-as-Code (IaC) approach. NeSI has an ongoing collaboration with AgResearch to build and operate an eResearch infrastructure. This was done on top of NeSI’s Flexible HPC, a platform designed to support flexibility and scale. Many of the learnings from this collaboration was applied to NeSI’s own infrastructure provisioning process, and much of the IaC based approach helped NeSI in taking the processes from the code base and modifying them to fit the needs. Through this journey, NeSI has matured greatly in its approach in code iteration and deployment pipelines.

The second approach is around NeSI’s data migration into the new infrastructure. NeSI has adopted new technologies, WEKA and Versity, to address the needs around ever growing data. WEKA is used for providing tiered active storage, and Versity for NeSI's new Freezer service, offering a long term archive solution. NeSI has applied thinking of data in temperature to allow smoother migration as well as ongoing HPC operations. This presentation will address many of the challenges around data migration and the solutions applied.

ABOUT THE AUTHORS

Jun Huh is a Product Manager at NeSI. Jun brings his experience from start-up industries into the field of eResearch. He has been involved in genomic data management related projects for the past 2-3 years, to help build a data repository system in partnership with Genomics Aotearoa, and more recently, prototyping for Rakeiora Pathfinder project, which focuses on enabling research while retaining full visibility and control of data to the indigenous communities.

Claire Rye is a Product Manager at New Zealand eScience Infrastructure (NeSI) based out of the University of Auckland. She is responsible for the National Data Transfer Service and works across the Aotearoa Genomics Data Repository and Rakeiora Pathfinder projects and looking at research data management and data lifecycle more generally across NeSI, and is a co-chair for the World Data System Early Career Researchers network. Claire holds a PhD in organic chemistry and has spent the last 11 years working in the UK in a variety of research settings.

Blair Bethwaite is NeSI’s Solutions Manager, when not on a bike, squash court or in the ocean, he specialises in innovation of the underlying systems and platforms to enable NeSI’s teams to deliver greater value to researchers. He has almost 20 years of experience in distributed computing; both in research and for research; for institutional and national projects; from science application collaborations, through grid & cloud middleware development, to full HPC & cloud system infrastructure design, implementation, and operations.

-----

For more information about the eResearch NZ / eRangahau Aotearoa conference, visit:
https://eresearchnz.co.nz/