In this talk I will give an account of a project we carried out to optimise and rework a pipeline to run at scale on a high performance computing facility while still remaining portable enough to run elsewhere. I’ll go into details of how we measure performance and scalability and go through the steps that were taken to lower the cost of running the pipeline. Finally, I’ll cover some of the problems that became apparent only when the pipeline went into production.
Senior HPC Specialist, National Computational Infrastructure (NCI)
Matthew Downton works in the Performance Optimisation team at NCI Australia. He has a background in computational science and high performance computing and has worked across a range of scientific disciplines. At NCI, he works together with user groups from across Australia, helping them make the most of the available computing facilities. Recently this has focused on software resources and workflows in bioinformatics.