To understand complex problems using machine learning it is generally necessary to have large amounts of data. In order to generate these large amounts of data, researchers utilize simulation. Simulations are best run on High-Performance Computers (HPCs) which require various complex processes. To simplify running machine learning based workflows on HPCs, LLNL scientists developed Merlin. The goal of Merlin is to make it easy to build, run, and process the kinds of large scale HPC workflows needed for cognitive simulation. At its heart, Merlin is a distributed task queuing system, designed to allow complex HPC workflows to scale to large numbers of simulations. The workflow software is applicable to any application space and uses other LLNL developed open source software, Maestro. Merlin has been used to study inertial confinement fusion, extreme ultraviolet light generation, structural mechanics and atomic physics, to name a few.
Merlin, open-sourced software licensed under the MIT license (LLNL internal case # CP02250)