escience2019 has ended
Back To Schedule
Thursday, September 26 • 3:30pm - 4:00pm
Dynamic Sizing of Continuously Divisible Jobs for Heterogeneous Resources

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Nicholas Hazekamp (University of Notre Dame), Benjamin Tovar (University of Notre Dame), and Douglas Thain (University of Notre Dame)

Many scientific applications operate on large datasets that can be partitioned and operated on concurrently. The existing approaches for concurrent execution generally rely on data which is statically partitioned. This static partitioning can lock performance in a sub-optimal configuration, leading to higher execution time and an inability to respond to dynamic resources.

We present the Continuously Divisible Job abstraction which allows statically defined applications to have their component tasks dynamically sized responding to system behaviour. The Continuously Divisible Job abstraction defines a simple interface that dictates how work can be recursively divided, executed, and merged. Implementing this abstraction allows scientific applications to leverage dynamic job coordinators for execution. We also propose the Virtual File abstraction which allows read-only subsets of large files to be treated as separate files.

In exploring the Continuously Divisible Job abstraction, two applications were implemented using the Continuously Divisible Job interface, a bioinformatics application and a high-energy physics event analysis. These were tested using an abstract job interface and several job coordinators. Comparing these against a previous static partitioning implementation we show comparable or better performance without having to make static decisions or implement complex dynamic application handling.


Nicholas Hazekamp

University of Notre Dame

Thursday September 26, 2019 3:30pm - 4:00pm PDT
Kon Tiki Room