Back To Schedule
Tuesday, July 9 • 3:00pm - 3:30pm
Caching in the Multiverse

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

To get good performance for data stored in Object storage services like S3, data analysis clusters need to cache data locally. Recently these caches have started taking into account higher-level information from analysis framework, allowing prefetching based on predictions of future data accesses. There is, however, a broader opportunity; rather than using this information to predict one future, we can use it to select a future that is best for caching. This paper provides preliminary evidence that we can exploit the directed acyclic graph (DAG) of inter-task dependencies used by data-parallel frameworks such as Spark, Pig, and Hive to improve application performance, by optimizing caching for the critical path through the DAG for the application. We present experimental results for PIG running TPC-H queries, showing completion time improvements of up to 23% vs our implementation of MRD, a state-of-the-art DAG-based prefetching system, and improvements of up to 2.5x vs LRU caching. We then discuss the broader opportunity for building a system based on this opportunity.


Mania Abdi

Northeastern University

Amin Mosayyebzadeh

Boston University

Mohammad Hossein Hajkazemi

Northeastern University

Ata Turk

State Street

Orran Krieger

Boston University

Peter Desnoyers

Northeastern University

Tuesday July 9, 2019 3:00pm - 3:30pm PDT
HotStorage: Grand Ballroom I–III