dlg.manager

This package contains all python modules implementing the DROP Manager concepts, including their external interface, a web UI and a client

dlg.manager.session

Module containing the logic of a session – a given graph execution

dlg.manager.drop_manager

Module containing the base interface for all DROP managers.

class dlg.manager.drop_manager.DROPManager

Base class for all DROPManagers.

A DROPManager, as the name states, manages the creation and execution of DROPs. In order to support parallel DROP graphs execution, a DROPManager separates them into “sessions”.

Sessions follow a simple lifecycle:

  • They are created in the PRISTINE status

  • One or more graph specifications are appended to them, which can also be linked together, building up the final graph specification. While building the graph the session is in the BUILDING status.

  • Once all graph specifications have been appended and linked together, the graph is deployed, meaning that the DROPs are effectively created. During this process the session transitions between the DEPLOYING and RUNNING states.

  • One all DROPs contained in a session have transitioned to COMPLETED (or ERROR, if there has been an error during the execution) the session moves to FINISHED.

Graph specifications are currently accepted in the form of a list of dictionaries, where each dictionary is a DROP specification. A DROP specification in turn consists on key/value pairs in the dictionary which state the type of DROP, some key parameters, and instance-specific parameters as well used to create the DROP.

abstract addGraphSpec(sessionId, graphSpec)

Adds a graph specification graphSpec (i.e., a description of the DROPs that should be created) to the current graph specification held by session sessionId.

abstract cancelSession(sessionId)

Cancels the session sessionId

abstract createSession(sessionId)

Creates a session on this DROPManager with id sessionId. A session represents an isolated DROP graph execution.

abstract deploySession(sessionId, completedDrops=[])

Deploys the graph specification held by session sessionId, effectively creating all DROPs, linking them together, and moving those whose UID is in completedDrops to the COMPLETED state.

abstract destroySession(sessionId)

Destroys the session sessionId

abstract getGraph(sessionId)

Returns a specification of the graph currently held by session sessionId.

abstract getGraphReproData(sessionId)

Returns the graph-wide reproducibility data for session sessionId

abstract getGraphSize(sessionId)

Returns the number of drops contained in the physical graph attached to sessionId.

abstract getGraphStatus(sessionId)

Returns the status of the graph being executed in session sessionId.

abstract getSessionIds()

Returns the IDs of the sessions currently held by this DROPManager.

abstract getSessionReproStatus(sessionId)

Returns the reproducibility status of the session sessionId. Not guaranteed to be identical to the usual SessionStatus.

abstract getSessionStatus(sessionId)

Returns the status of the session sessionId.

dlg.manager.node_manager

Module containing the NodeManager, which directly manages DROP instances, and thus represents the bottom of the DROP management hierarchy.

class dlg.manager.node_manager.ErrorStatusListener(session, error_listener)

An event listener that passes down the erroneous drop to an error handler

dlg.manager.node_manager.EventMixIn

alias of ZMQPubSubMixIn

class dlg.manager.node_manager.NodeManager(host=None, rpc_port=6666, events_port=5555, *args, **kwargs)
class dlg.manager.node_manager.NodeManagerBase(dlm_check_period=0, dlm_cleanup_period=0, dlm_enable_replication=False, dlgPath=None, error_listener=None, event_listeners=[], max_threads=0, logdir='/home/docs/dlg/logs')

Base class for a DROPManager that creates and holds references to DROPs.

A NodeManagerBase is the ultimate responsible of handling DROPs. It does so not directly, but via Sessions, which represent and encapsulate separate, independent DROP graph executions. All DROPs created by the different Sessions are also given to a common DataLifecycleManager, which takes care of expiring them when needed and replicating them.

Since a NodeManagerBase can handle more than one session, in principle only one NodeManagerBase is needed for each computing node, thus its name.

addGraphSpec(sessionId, graphSpec)

Adds a graph specification graphSpec (i.e., a description of the DROPs that should be created) to the current graph specification held by session sessionId.

cancelSession(sessionId)

Cancels the session sessionId

createSession(sessionId)

Creates a session on this DROPManager with id sessionId. A session represents an isolated DROP graph execution.

deliver_event(evt)

Method called by subclasses when a new event has arrived through the subscription mechanism.

deploySession(sessionId, completedDrops=[])

Deploys the graph specification held by session sessionId, effectively creating all DROPs, linking them together, and moving those whose UID is in completedDrops to the COMPLETED state.

destroySession(sessionId)

Destroys the session sessionId

getGraph(sessionId)

Returns a specification of the graph currently held by session sessionId.

getGraphReproData(sessionId)

Returns the graph-wide reproducibility data for session sessionId

getGraphSize(sessionId)

Returns the number of drops contained in the physical graph attached to sessionId.

getGraphStatus(sessionId)

Returns the status of the graph being executed in session sessionId.

getSessionIds()

Returns the IDs of the sessions currently held by this DROPManager.

getSessionReproStatus(sessionId)

Returns the reproducibility status of the session sessionId. Not guaranteed to be identical to the usual SessionStatus.

getSessionStatus(sessionId)

Returns the status of the session sessionId.

class dlg.manager.node_manager.RpcMixIn(*args, **kwargs)
class dlg.manager.node_manager.ZMQPubSubMixIn(host, events_port)

ZeroMQ-based event publisher and subscriber.

Event publishing and event reception are done in their own separate threads, where the externally-facing ZeroMQ sockets are created and used.

Events to be published are fed into the publishing thread via a safe-thread Queue object (self._events_out), enabling any local thread to publish events without having to worry about ZeroMQ thread-safeness.

The event reception thread not only receives events, but also updates the subscription socket to connect to new peers. These updates are fed via a Queue object (self._subscriptions), enabling any local thread to indicate a new peer to subscribe to in a thread-safe manner.

Note that we investigated not using Queue objects to communicate between threads, and use inproc:// ZeroMQ sockets instead. This works, but at a cost: all threads putting values into these sockets would need to check, each time they use a socket in any manner, if the Context object is still valid and hasn’t been closed (or alternatively if self._pubsub_running is still True). Our experience with this alternative was not satisfactory, and therefore we went for a Queue-based thread communication model, making the handling of ZeroMQ resources simpler.

class subscription(endpoint, finished_evt)
property endpoint

Alias for field number 0

property finished_evt

Alias for field number 1

dlg.manager.composite_manager

class dlg.manager.composite_manager.CompositeManager(dmPort, partitionAttr, subDmId, dmHosts=[], pkeyPath=None, dmCheckTimeout=10)

A DROPManager that in turn manages DROPManagers (sigh…).

DROP Managers form a hierarchy where those at the bottom actually hold DROPs while those in the levels above rely commands and aggregate results, making the system more manageable and scalable. The CompositeManager class implements the upper part of this hierarchy in a generic way by holding references to a number of sub-DROPManagers and communicating with them to complete each operation. The only assumption about sub-DROPManagers is that they obey the DROPManager interface, and therefore this CompositeManager class allows for multiple levels of hierarchy seamlessly.

Having different levels of Data Management hierarchy implies that the physical graph that is fed into the hierarchy needs to be partitioned at each level (except at the bottom of the hierarchy) in order to place each DROP in its correct place. The attribute used by a particular CompositeManager to partition the graph (from its graphSpec) is given at construction time.

addGraphSpec(sessionId, graphSpec)

Adds a graph specification graphSpec (i.e., a description of the DROPs that should be created) to the current graph specification held by session sessionId.

cancelSession(sessionId)

Cancels a session in all underlying DMs.

createSession(sessionId)

Creates a session in all underlying DMs.

deploySession(sessionId, completedDrops=[])

Deploys the graph specification held by session sessionId, effectively creating all DROPs, linking them together, and moving those whose UID is in completedDrops to the COMPLETED state.

destroySession(sessionId)

Destroy a session in all underlying DMs.

getGraph(sessionId)

Returns a specification of the graph currently held by session sessionId.

getGraphSize(sessionId)

Returns the number of drops contained in the physical graph attached to sessionId.

getGraphStatus(sessionId)

Returns the status of the graph being executed in session sessionId.

getSessionIds()

Returns the IDs of the sessions currently held by this DROPManager.

getSessionStatus(sessionId)

Returns the status of the session sessionId.

replicate(sessionId, f, action, collect=None, iterable=None, port=None)

Replicates the given function call on each of the underlying drop managers

class dlg.manager.composite_manager.DataIslandManager(dmHosts=[], pkeyPath=None, dmCheckTimeout=10)

The DataIslandManager, which manages a number of NodeManagers.

class dlg.manager.composite_manager.MasterManager(dmHosts=[], pkeyPath=None, dmCheckTimeout=10)

The MasterManager, which manages a number of DataIslandManagers.

Links can now be dictionaries, but we only need the key.

dlg.manager.rest

Module containing the REST layer that exposes the methods of the different Data Managers (DROPManager and DataIslandManager) to the outside world.

class dlg.manager.rest.CompositeManagerRestServer(dm, maxreqsize=10)

A REST server for DataIslandManagers. It includes mappings for DIM-specific methods.

initializeSpecifics(app)

Methods through which subclasses can initialize other mappings on top of the default ones and perform other DataManager-specific actions. The default implementation does nothing.

class dlg.manager.rest.ManagerRestServer(dm, maxreqsize=10)

An object that wraps a DataManager and exposes its methods via a REST interface. The server is started via the start method in a separate thread and runs until the process is shut down.

This REST server currently also serves HTML pages in some of its methods (i.e. those not under /api).

initializeSpecifics(app)

Methods through which subclasses can initialize other mappings on top of the default ones and perform other DataManager-specific actions. The default implementation does nothing.

class dlg.manager.rest.MasterManagerRestServer(dm, maxreqsize=10)
initializeSpecifics(app)

Methods through which subclasses can initialize other mappings on top of the default ones and perform other DataManager-specific actions. The default implementation does nothing.

class dlg.manager.rest.NMRestServer(dm, maxreqsize=10)

A REST server for NodeManagers. It includes mappings for NM-specific methods and the mapping for the main visualization HTML pages.

initializeSpecifics(app)

Methods through which subclasses can initialize other mappings on top of the default ones and perform other DataManager-specific actions. The default implementation does nothing.

dlg.manager.client

Backwards compatibility for client