dlg.manager

This package contains all python modules implementing the DROP Manager concepts, including their external interface, a web UI and a client

dlg.manager.session

Module containing the logic of a session – a given graph execution

dlg.manager.drop_manager

Module containing the base interface for all DROP managers.

class dlg.manager.drop_manager.DROPManager

Base class for all DROPManagers.

A DROPManager, as the name states, manages the creation and execution of DROPs. In order to support parallel DROP graphs execution, a DROPManager separates them into “sessions”.

Sessions follow a simple lifecycle:
  • They are created in the PRISTINE status
  • One or more graph specifications are appended to them, which can also be linked together, building up the final graph specification. While building the graph the session is in the BUILDING status.
  • Once all graph specifications have been appended and linked together, the graph is deployed, meaning that the DROPs are effectively created. During this process the session transitions between the DEPLOYING and RUNNING states.
  • One all DROPs contained in a session have transitioned to COMPLETED (or ERROR, if there has been an error during the execution) the session moves to FINISHED.

Graph specifications are currently accepted in the form of a list of dictionaries, where each dictionary is a DROP specification. A DROP specification in turn consists on key/value pairs in the dictionary which state the type of DROP, some key parameters, and instance-specific parameters as well used to create the DROP.

addGraphSpec(sessionId, graphSpec)

Adds a graph specification graphSpec (i.e., a description of the DROPs that should be created) to the current graph specification held by session sessionId.

createSession(sessionId)

Creates a session on this DROPManager with id sessionId. A session represents an isolated DROP graph execution.

deploySession(sessionId, completedDrops=[])

Deploys the graph specification held by session sessionId, effectively creating all DROPs, linking them together, and moving those whose UID is in completedDrops to the COMPLETED state.

destroySession(sessionId)

Destroys the session sessionId

getGraph(sessionId)

Returns a specification of the graph currently held by session sessionId.

getGraphSize(sessionId)

Returns the number of drops contained in the physical graph attached to sessionId.

getGraphStatus(sessionId)

Returns the status of the graph being executed in session sessionId.

getSessionIds()

Returns the IDs of the sessions currently held by this DROPManager.

getSessionStatus(sessionId)

Returns the status of the session sessionId.

dlg.manager.node_manager

Module containing the NodeManager, which directly manages DROP instances, and thus represents the bottom of the DROP management hierarchy.

class dlg.manager.node_manager.ErrorStatusListener(session, error_listener)

An event listener that passes down the erroneous drop to an error handler

class dlg.manager.node_manager.NodeManager(useDLM=True, dlgPath=None, error_listener=None, event_listeners=[], max_threads=0, host=None, rpc_port=6666, events_port=5555)
class dlg.manager.node_manager.NodeManagerBase(useDLM=True, dlgPath=None, error_listener=None, event_listeners=[], max_threads=0)

Base class for a DROPManager that creates and holds references to DROPs.

A NodeManagerBase is the ultimate responsible of handling DROPs. It does so not directly, but via Sessions, which represent and encapsulate separate, independent DROP graph executions. All DROPs created by the different Sessions are also given to a common DataLifecycleManager, which takes care of expiring them when needed and replicating them.

Since a NodeManagerBase can handle more than one session, in principle only one NodeManagerBase is needed for each computing node, thus its name.

addGraphSpec(sessionId, graphSpec)

Adds a graph specification graphSpec (i.e., a description of the DROPs that should be created) to the current graph specification held by session sessionId.

createSession(sessionId)

Creates a session on this DROPManager with id sessionId. A session represents an isolated DROP graph execution.

deliver_event(evt)

Method called by subclasses when a new event has arrived through the subscription mechanism.

deploySession(sessionId, completedDrops=[])

Deploys the graph specification held by session sessionId, effectively creating all DROPs, linking them together, and moving those whose UID is in completedDrops to the COMPLETED state.

destroySession(sessionId)

Destroys the session sessionId

getGraph(sessionId)

Returns a specification of the graph currently held by session sessionId.

getGraphSize(sessionId)

Returns the number of drops contained in the physical graph attached to sessionId.

getGraphStatus(sessionId)

Returns the status of the graph being executed in session sessionId.

getSessionIds()

Returns the IDs of the sessions currently held by this DROPManager.

getSessionStatus(sessionId)

Returns the status of the session sessionId.

get_rpc_client(hostname, port)

Creates an RPC client connected to the node manager running in host:port, and its closing method, as a 2-tuple.

publish_event(evt)

Publishes the event evt for other Node Managers to receive it

shutdown()

Stops any pending background task run by this Node Manager

start()

Starts any background task required by this Node Manager

subscribe(host, port)

Subscribes this Node Manager to events published in from host:port

class dlg.manager.node_manager.RpcMixIn(host, port)

dlg.manager.rest

Module containing the REST layer that exposes the methods of the different Data Managers (DROPManager and DataIslandManager) to the outside world.

class dlg.manager.rest.CompositeManagerRestServer(dm, maxreqsize=10)

A REST server for DataIslandManagers. It includes mappings for DIM-specific methods.

initializeSpecifics(app)

Methods through which subclasses can initialize other mappings on top of the default ones and perform other DataManager-specific actions. The default implementation does nothing.

class dlg.manager.rest.ManagerRestServer(dm, maxreqsize=10)

An object that wraps a DataManager and exposes its methods via a REST interface. The server is started via the start method in a separate thread and runs until the process is shut down.

This REST server currently also serves HTML pages in some of its methods (i.e. those not under /api).

initializeSpecifics(app)

Methods through which subclasses can initialize other mappings on top of the default ones and perform other DataManager-specific actions. The default implementation does nothing.

class dlg.manager.rest.MasterManagerRestServer(dm, maxreqsize=10)
initializeSpecifics(app)

Methods through which subclasses can initialize other mappings on top of the default ones and perform other DataManager-specific actions. The default implementation does nothing.

class dlg.manager.rest.NMRestServer(dm, maxreqsize=10)

A REST server for NodeManagers. It includes mappings for NM-specific methods and the mapping for the main visualization HTML pages.

initializeSpecifics(app)

Methods through which subclasses can initialize other mappings on top of the default ones and perform other DataManager-specific actions. The default implementation does nothing.

dlg.manager.client

class dlg.manager.client.BaseDROPManagerClient(host, port, timeout)

Base class for REST clients that talk to the DROP managers.

addGraphSpec(sessionId, graphSpec)

Appends a graph to session sessionId, without creating its DROPs yet, but checking that the graph looks correct

append_graph(sessionId, graphSpec)

Appends a graph to session sessionId, without creating its DROPs yet, but checking that the graph looks correct

createSession(sessionId)

Creates a session with sessionId

create_session(sessionId)

Creates a session with sessionId

deploySession(sessionId, completed_uids=[])

Deploys session sessionId, effectively creating its DROPs and triggering the execution of the graph

deploy_session(sessionId, completed_uids=[])

Deploys session sessionId, effectively creating its DROPs and triggering the execution of the graph

destroySession(sessionId)

Destroys session sessionId

destroy_session(sessionId)

Destroys session sessionId

getGraph(sessionId)

Returns a dictionary where the key are the DROP UIDs, and the values are the DROP specifications.

getGraphSize(sessionId)

Returns the size of the graph of session sessionId

getGraphStatus(sessionId)

Returns a dictionary where the keys are DROP UIDs and the values are their corresponding status.

getSessionStatus(sessionId)

Returns the status of session sessionId

graph(sessionId)

Returns a dictionary where the key are the DROP UIDs, and the values are the DROP specifications.

graph_size(sessionId)

Returns the size of the graph of session sessionId

graph_status(sessionId)

Returns a dictionary where the keys are DROP UIDs and the values are their corresponding status.

session(sessionId)

Returns the details of sessions sessionId

session_status(sessionId)

Returns the status of session sessionId

sessions()

Returns a list of all the sessions currently held by the DROP Manager

class dlg.manager.client.CompositeManagerClient(host, port, timeout)
class dlg.manager.client.DataIslandManagerClient(host='localhost', port=8001, timeout=10)

A DataIslandManager REST client

class dlg.manager.client.MasterManagerClient(host='localhost', port=8002, timeout=10)

A MasterManager REST client

class dlg.manager.client.NodeManagerClient(host='localhost', port=8000, timeout=10)

A NodeManager REST client