DALiuGE

Welcome to the Data Activated 流 1 Graph Engine (DALiuGE).

DALiuGE is a workflow graph execution framework, specifically designed to support very large scale processing graphs for the reduction of interferometric radio astronomy data sets. DALiuGE has already been used for processing large astronomical datasets in existing radio astronomy projects. It originated from a prototyping activity as part of the SDP Consortium called Data Flow Management System (DFMS). DFMS aimed to prototype the execution framework of the proposed SDP architecture. For a complete tour of DALiuGE please read our overview paper. DALiuGE has been used in a project running a full-scale simulation of the Square Kilometre Array dataflow on the ORNL Summit supercomputer.

Development and maintenance of DALiuGE is currently hosted at ICRAR and is performed by the DIA team.

• Introduction
• Installation Guide
  • Docker images
  • Direct Installation
  • Notes of the merge project between DALiuGE and Ray
• Startup and Shutdown Guide
  • For the impatient: Single node DALiuGE
  • Starting the docker containers
  • Starting and stopping using CLI
  • For the independent: Build and run EAGLE
  • Zeroconf
• Basics
  • EAGLE
  • Translator service
  • Execution Engine
• Architecture and Design
  • Concepts and Background
  • Drops
  • Graphs
  • Drop Managers
  • Data Lifecycle Manager
  • Scientific Reproducibility
  • References
• Operational concepts
  • Deployment Scenarios
  • Component Deployment
• Graph development
  • Using the Logical Graph Editor EAGLE
  • Directly creating a Physical Graph
  • Using dlg.delayed()
• DALiuGE Component Developers Guide
  • Introduction to Component development
  • Template Primer
  • DALiuGE Application Component Developers Guide
  • DALiuGE Data Component Developers Guide
  • Citations
• DALiuGE Users guide
  • Users Guide Introduction
  • Data Reduction User’s Guide
  • Graph Developer’s Guide
• CLI User Guide
  • Basic Usage
  • Subcommand usage
• API Documentation
  • dlg
  • dlg.manager
  • dlg.apps
  • dlg.data
  • dlg.dropmake

Citations

As you use DALiuGE for your exciting projects, please cite the following paper:

Wu, C., Tobar, R., Vinsen, K., Wicenec, A., Pallot, D., Lao, B., Wang, R., An, T., Boulton, M., Cooper, I. and Dodson, R., 2017. DALiuGE: A Graph Execution Framework for Harnessing the Astronomical Data Deluge. Astronomy and Computing, 20, pp.1-15. (2017)

1

流 (pronounced Liu) is the Chinese character for “flow”.