DALiuGE

Welcome to the Data Activated 流 [1] Graph Engine (DALiuGE).

DALiuGE is a workflow graph execution framework, specifically designed to support very large scale processing graphs for the reduction of interferometric radio astronomy data sets. DALiuGE has already been used for processing large astronomical datasets in existing radio astronomy projects. It originated from a prototyping activity as part of the SDP Consortium called Data Flow Management System (DFMS). DFMS aimed to prototype the execution framework of the proposed SDP architecture. For a complete tour of DALiuGE please read our overview paper. DALiuGE has been used in a project running a full-scale simulation of the Square Kilometre Array dataflow on the ORNL Summit supercomputer.

_images/DALiuGE_naming_rationale.png

Development and maintenance of DALiuGE is currently hosted at ICRAR and is performed by the DIA team.

Quick Start Guide

Introduction
Installation
Running DALiuGE
Hello, World

Graph Patterns

Loop

scatter branch subgraph globals

Engine Fundamentals

Architecture and Design
Operational Concepts

Using the command line interface

Basics
Engine Commands
Translator Commands

Advanced DALiuGE Methods

Delayed Execution

Debugging

Error Codes

API Documentation

API Documentation
Application Component Developers Guide
Data Component Developers Guide

Citations

As you use DALiuGE for your exciting projects, please cite the following paper:

Wu, C., Tobar, R., Vinsen, K., Wicenec, A., Pallot, D., Lao, B., Wang, R., An, T., Boulton, M., Cooper, I. and Dodson, R., 2017. DALiuGE: A Graph Execution Framework for Harnessing the Astronomical Data Deluge. Astronomy and Computing, 20, pp.1-15. (2017)