escience2019 has ended
Friday, September 27 • 12:00pm - 12:30pm
Toward an Elastic Data Transfer Infrastructure

Sign up or log in to save this to your schedule and see who's attending!

Joaquin Chung (Argonne National Laboratory), Zhengchun Liu (Argonne National Laboratory), Rajkumar Kettimuthu (Argonne National Laboratory), and Ian Foster (Argonne National Laboratory)

Data transfer over wide area network is an integral part of many science workflows. These workflows move data produced at experimental, observational and/or computational facilities to geographically distributed resources for analysis, sharing, and storing. Enhancements in the data transfer infrastructure of universities have improved the performance of data transfers for a number of users and science workflows. Despite these positive developments, our previous analyses of approximately 40 billion GridFTP command logs totaling 3.3 exabytes and 4.8 million transfers logs collected by the Globus transfer service from 2014/01/01 to 2018/01/01 show that data transfer nodes (DTNs) are completely idle (i.e., no transfers) 94.3% of the time. Furthermore, 80% of the DTNs are active less than 6% of the time. Motivated by the opportunity to optimize the architecture of data transfer infrastructure, we developed an elastic architecture for data transfer in which the DTNs expand and shrink based on the demand. Our architecture is composed of agents that monitor resource utilization at bare metal nodes and an orchestrator that decides when to provision/deprovision resources. Our results show that our elastic DTI can save up to 98% of resources compared with a typical DTN deployment, while experiencing only minimal overhead (∼1%).


Rajkumar Kettimuthu

Argonne National Laboratory

Friday September 27, 2019 12:00pm - 12:30pm
Cockatoo Room