Overview

We study workflow technologies and frameworks for big data processing and management

  • Slides of the lecture

Reading List

  •  [Running Apache Airflow at Lyft](https://eng.lyft.com/running-apache-airflow-at-lyft-6e53bb8fccff)
  • Mutaz Barika, Saurabh Garg, Albert Y. Zomaya, Lizhe Wang, Aad Van Moorsel, and Rajiv Ranjan. 2019. Orchestrating Big Data Analysis Workflows in the Cloud: Research Challenges, Survey, and Future Directions. ACM Comput. Surv. 52, 5, Article 95 (September 2019), 41 pages. DOI: https://doi.org/10.1145/3332301
  • [How Agari Uses Airbnb's Airflow as a Smarter Cron](http://highscalability.com/blog/2015/9/3/how-agari-uses-airbnbs-airflow-as-a-smarter-cron.html)
  • Ewa Deelman, Karan Vahi, Mats Rynge, Rajiv Mayani, Rafael Ferreira da Silva, George Papadimitriou, Miron Livny: The Evolution of the Pegasus Workflow Management Software. Computing in Science and Engineering 21(4): 22-36 (2019)
  • Mohammad Islam, Angelo K. Huang, Mohamed Battisha, Michelle Chiang, Santhosh Srinivasan, Craig Peters, Andreas Neumann, and Alejandro Abdelnur. 2012. Oozie: towards a scalable workflow management system for Hadoop. In Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies (SWEET '12). ACM, New York, NY, USA, Article 4, 10 pages. DOI: https://doi.org/10.1145/2443416.2443420
  • Ian J. Taylor, Ewa Deelman, Dennis B. Gannon, and Matthew Shields. 2006. Workflows for E-Science: Scientific Workflows for Grids. Springer-Verlag, Berlin, Heidelberg.
Senast redigerad: onsdag, 19 januari 2022, 14:49