Installation

Note

This guide is meant for people who are developing and testing workflows using EAGLE, or who are experimenting with the system. It does not cover specific needs of more complex, distributed operational deployments.

DALiuGE applications

For development and testing of workflows for DALiuGE, the following runtime environments are necessary:

• daliuge-translator: This takes the workflow from EAGLE and ‘unrolls’ it into the workflow DAG that will be executed (see Translation).

• daliuge-engine: This manages workflow execution across many nodes. There is a hierarchy of applications within the engine that must be run to manage the execution at different level of granularity (i.e. at the node level, the data-island level, and at the master level; see Drop Managers).

The intricacies of the different environments and their applications will be covered in more detail in later sections of the documentation. For the purpose of starting out, it is enough to know that to start developing and testing workflows on DALiuGE, the following is necessary:

• EAGLE workflow has a graph that is ready to translate;

• daliuge-translator is running, waiting to receive a workflow from EAGLE; and

• daliuge-engine applications (A Data Island Manager and a Node Manager) are running.

The following instructions explain how to set up your environment to achieve this.

Installation Options

I am:	This involves:	Best option:
Developing and testing graphs locally (Best option for first time users.)	Using EAGLE in the browser Translate workflows through the browser Deploy workflows locally	Docker Installation
Developing and deploying graphs that use an external library (e.g. numpy)	Developing EAGLE graphs in the browser Translating through the browser or CLI Using the dlg CLI Deploying through the browser or CLI	PyPI Installation
Adminstrator of a HPC cluster interested in DALiuGE	Using the dlg CLI Developing EAGLE graphs in the browser Translating through the browser or CLI Deploying through the browser or CLI	PyPI Installation
Contributing to DALiuGE software	Writing and testing the software Wanting the latest features Iterating on local source code	Direct Installation

Docker Installation

Recommended

Note

This assumes docker is installed locally on your system before running these commands. Please review https://docs.docker.com/get-started/get-docker/ if Docker is not installed on your system.

The recommended and easiest way to get started is to use the docker containers for the daliuge-engine and daliuge-translator. To download the images for the purpose of developing workflows with the EAGLE editor, the following is all that is necessary:

docker pull icrar/daliuge-engine.slim:latest
docker pull icrar/daliuge-translator.slim:latest

These are minified docker images that contain only the necessary material for running the applications. To start running the applications, head straight to DALiuGE in Docker.

Alternative: Building the images locally

This section assumes you have experience using Git and Docker, and want to build from the (potentially unstable) DALiuGE source code.

First, clone the DALiuGE github repository:

git clone https://github.com/ICRAR/daliuge
cd daliuge

Then using the make utility we build the docker images for a development environment:

make docker-install

This will install daliuge-common, daliuge-engine, and daliuge-translator based on the local development state of the DALiuGE codebase.

Building images manually

Building the three manuall is easy, just start with the daliuge-common image by running:

cd daliuge-common && ./build_common.sh dev && cd ..

then build the runtime:

cd daliuge-engine&& ./build_engine.sh dev && cd ..

and last build the translator:

cd daliuge-translator && ./build_translator.sh dev && cd ..

Running the images

Running the development version of the engine and the translator follows the same logic as above:

make docker-run

You can use EAGLE on the URL: https://eagle.icrar.org and point your EAGLE configuration for the translator to http://dlg-tm.localhost. Congratulations! You now have access to a complete DALiuGE system on your local computer!

More detailed information about running and controlling the DALiuGE system can be found in the Running DALiuGE.

Running images manually

Running the engine and the translator is equally simple:

cd daliuge-engine && ./run_engine.sh dev && cd ..

and:

cd daliuge-translator && ./run_translator.sh dev && cd ..

PyPI Installation

Note

DALiuGE requires python 3.9 or later. It is always recommended to install DALiuGE inside it’s own Python virtual environment. Make sure that you have on created and enabled. More often than not pip requries an update, else it will always issue a warning:

pip install --upgrade pip

Inside your virtual environment, the latest version of the DALiuGE can be installed as follows:

pip install daliuge-common && pip install daliuge-engine && pip install daliuge-translator

DALiuGE may now be run using the CLI interface.

Updating PyPI installation

An existing installation can be updated using:

pip install -U daliuge-common && pip install -U daliuge-engine && pip install -U daliuge-translato

Direct Installation

Note

DALiuGE requires python 3.9 or later.

It is always recommended to install DALiuGE inside it’s own Python virtual environment. Make sure that you have on created and enabled. More often than not pip requries an update, else it will always issue a warning:

pip install --upgrade pip

First, clone the DALiuGE github repository:

git clone https://github.com/ICRAR/daliuge
cd daliuge

Perform the following steps to setup and install DALiuGE into the specific virtual environment:

cd daliuge
# source your virtual env
make show # Optional, use to confirm virtualenv is active
make install

Installing from source manually

After cloning and entering the daliuge folder:

cd daliuge-common
pip install -e .
cd daliuge-engine
pip install -e .
cd daliuge-translator
pip install -e .

Alternative installation options

Installing from GitHub

The following commands are installing the DALiuGE parts directly from github. In this case you won’t have access to the sources, but the system will run. First install the daliuge-common part:

pip install 'git+https://github.com/ICRAR/daliuge.git#egg&subdirectory=daliuge-common'

then install the daliuge-engine:

pip install 'git+https://github.com/ICRAR/daliuge.git#egg&subdirectory=daliuge-engine'

and finally, if required also install the daliuge-translator:

pip install 'git+https://github.com/ICRAR/daliuge.git#egg&subdirectory=daliuge-translator'