dlg.manager¶
This package contains all python modules implementing the DROP Manager concepts, including their external interface, a web UI and a client
Contents
dlg.manager.session¶
Module containing the logic of a session – a given graph execution
dlg.manager.drop_manager¶
Module containing the base interface for all DROP managers.
-
class
dlg.manager.drop_manager.
DROPManager
¶ Base class for all DROPManagers.
A DROPManager, as the name states, manages the creation and execution of DROPs. In order to support parallel DROP graphs execution, a DROPManager separates them into “sessions”.
- Sessions follow a simple lifecycle:
- They are created in the PRISTINE status
- One or more graph specifications are appended to them, which can also be linked together, building up the final graph specification. While building the graph the session is in the BUILDING status.
- Once all graph specifications have been appended and linked together, the graph is deployed, meaning that the DROPs are effectively created. During this process the session transitions between the DEPLOYING and RUNNING states.
- One all DROPs contained in a session have transitioned to COMPLETED (or ERROR, if there has been an error during the execution) the session moves to FINISHED.
Graph specifications are currently accepted in the form of a list of dictionaries, where each dictionary is a DROP specification. A DROP specification in turn consists on key/value pairs in the dictionary which state the type of DROP, some key parameters, and instance-specific parameters as well used to create the DROP.
-
addGraphSpec
(sessionId, graphSpec)¶ Adds a graph specification graphSpec (i.e., a description of the DROPs that should be created) to the current graph specification held by session sessionId.
-
createSession
(sessionId)¶ Creates a session on this DROPManager with id sessionId. A session represents an isolated DROP graph execution.
-
deploySession
(sessionId, completedDrops=[])¶ Deploys the graph specification held by session sessionId, effectively creating all DROPs, linking them together, and moving those whose UID is in completedDrops to the COMPLETED state.
-
destroySession
(sessionId)¶ Destroys the session sessionId
-
getGraph
(sessionId)¶ Returns a specification of the graph currently held by session sessionId.
-
getGraphSize
(sessionId)¶ Returns the number of drops contained in the physical graph attached to
sessionId
.
-
getGraphStatus
(sessionId)¶ Returns the status of the graph being executed in session sessionId.
-
getSessionIds
()¶ Returns the IDs of the sessions currently held by this DROPManager.
-
getSessionStatus
(sessionId)¶ Returns the status of the session sessionId.
dlg.manager.node_manager¶
Module containing the NodeManager, which directly manages DROP instances, and thus represents the bottom of the DROP management hierarchy.
-
class
dlg.manager.node_manager.
ErrorStatusListener
(session, error_listener)¶ An event listener that passes down the erroneous drop to an error handler
-
class
dlg.manager.node_manager.
NodeManager
(useDLM=True, dlgPath=None, error_listener=None, event_listeners=[], max_threads=0, host=None, rpc_port=6666, events_port=5555)¶
-
class
dlg.manager.node_manager.
NodeManagerBase
(useDLM=True, dlgPath=None, error_listener=None, event_listeners=[], max_threads=0)¶ Base class for a DROPManager that creates and holds references to DROPs.
A NodeManagerBase is the ultimate responsible of handling DROPs. It does so not directly, but via Sessions, which represent and encapsulate separate, independent DROP graph executions. All DROPs created by the different Sessions are also given to a common DataLifecycleManager, which takes care of expiring them when needed and replicating them.
Since a NodeManagerBase can handle more than one session, in principle only one NodeManagerBase is needed for each computing node, thus its name.
-
addGraphSpec
(sessionId, graphSpec)¶ Adds a graph specification graphSpec (i.e., a description of the DROPs that should be created) to the current graph specification held by session sessionId.
-
createSession
(sessionId)¶ Creates a session on this DROPManager with id sessionId. A session represents an isolated DROP graph execution.
-
deliver_event
(evt)¶ Method called by subclasses when a new event has arrived through the subscription mechanism.
-
deploySession
(sessionId, completedDrops=[])¶ Deploys the graph specification held by session sessionId, effectively creating all DROPs, linking them together, and moving those whose UID is in completedDrops to the COMPLETED state.
-
destroySession
(sessionId)¶ Destroys the session sessionId
-
getGraph
(sessionId)¶ Returns a specification of the graph currently held by session sessionId.
-
getGraphSize
(sessionId)¶ Returns the number of drops contained in the physical graph attached to
sessionId
.
-
getGraphStatus
(sessionId)¶ Returns the status of the graph being executed in session sessionId.
-
getSessionIds
()¶ Returns the IDs of the sessions currently held by this DROPManager.
-
getSessionStatus
(sessionId)¶ Returns the status of the session sessionId.
-
get_rpc_client
(hostname, port)¶ Creates an RPC client connected to the node manager running in
host
:port
, and its closing method, as a 2-tuple.
-
publish_event
(evt)¶ Publishes the event
evt
for other Node Managers to receive it
-
shutdown
()¶ Stops any pending background task run by this Node Manager
-
start
()¶ Starts any background task required by this Node Manager
-
subscribe
(host, port)¶ Subscribes this Node Manager to events published in from
host
:port
-
-
class
dlg.manager.node_manager.
RpcMixIn
(host, port)¶
dlg.manager.rest¶
Module containing the REST layer that exposes the methods of the different Data Managers (DROPManager and DataIslandManager) to the outside world.
-
class
dlg.manager.rest.
CompositeManagerRestServer
(dm, maxreqsize=10)¶ A REST server for DataIslandManagers. It includes mappings for DIM-specific methods.
-
initializeSpecifics
(app)¶ Methods through which subclasses can initialize other mappings on top of the default ones and perform other DataManager-specific actions. The default implementation does nothing.
-
-
class
dlg.manager.rest.
ManagerRestServer
(dm, maxreqsize=10)¶ An object that wraps a DataManager and exposes its methods via a REST interface. The server is started via the start method in a separate thread and runs until the process is shut down.
This REST server currently also serves HTML pages in some of its methods (i.e. those not under /api).
-
initializeSpecifics
(app)¶ Methods through which subclasses can initialize other mappings on top of the default ones and perform other DataManager-specific actions. The default implementation does nothing.
-
-
class
dlg.manager.rest.
MasterManagerRestServer
(dm, maxreqsize=10)¶ -
initializeSpecifics
(app)¶ Methods through which subclasses can initialize other mappings on top of the default ones and perform other DataManager-specific actions. The default implementation does nothing.
-
-
class
dlg.manager.rest.
NMRestServer
(dm, maxreqsize=10)¶ A REST server for NodeManagers. It includes mappings for NM-specific methods and the mapping for the main visualization HTML pages.
-
initializeSpecifics
(app)¶ Methods through which subclasses can initialize other mappings on top of the default ones and perform other DataManager-specific actions. The default implementation does nothing.
-
dlg.manager.client¶
-
class
dlg.manager.client.
BaseDROPManagerClient
(host, port, timeout)¶ Base class for REST clients that talk to the DROP managers.
-
addGraphSpec
(sessionId, graphSpec)¶ Appends a graph to session sessionId, without creating its DROPs yet, but checking that the graph looks correct
-
append_graph
(sessionId, graphSpec)¶ Appends a graph to session sessionId, without creating its DROPs yet, but checking that the graph looks correct
-
createSession
(sessionId)¶ Creates a session with sessionId
-
create_session
(sessionId)¶ Creates a session with sessionId
-
deploySession
(sessionId, completed_uids=[])¶ Deploys session sessionId, effectively creating its DROPs and triggering the execution of the graph
-
deploy_session
(sessionId, completed_uids=[])¶ Deploys session sessionId, effectively creating its DROPs and triggering the execution of the graph
-
destroySession
(sessionId)¶ Destroys session sessionId
-
destroy_session
(sessionId)¶ Destroys session sessionId
-
getGraph
(sessionId)¶ Returns a dictionary where the key are the DROP UIDs, and the values are the DROP specifications.
-
getGraphSize
(sessionId)¶ Returns the size of the graph of session sessionId
-
getGraphStatus
(sessionId)¶ Returns a dictionary where the keys are DROP UIDs and the values are their corresponding status.
-
getSessionStatus
(sessionId)¶ Returns the status of session sessionId
-
graph
(sessionId)¶ Returns a dictionary where the key are the DROP UIDs, and the values are the DROP specifications.
-
graph_size
(sessionId)¶ Returns the size of the graph of session sessionId
-
graph_status
(sessionId)¶ Returns a dictionary where the keys are DROP UIDs and the values are their corresponding status.
-
session
(sessionId)¶ Returns the details of sessions sessionId
-
session_status
(sessionId)¶ Returns the status of session sessionId
-
sessions
()¶ Returns a list of all the sessions currently held by the DROP Manager
-
-
class
dlg.manager.client.
CompositeManagerClient
(host, port, timeout)¶
-
class
dlg.manager.client.
DataIslandManagerClient
(host='localhost', port=8001, timeout=10)¶ A DataIslandManager REST client
-
class
dlg.manager.client.
MasterManagerClient
(host='localhost', port=8002, timeout=10)¶ A MasterManager REST client
-
class
dlg.manager.client.
NodeManagerClient
(host='localhost', port=8000, timeout=10)¶ A NodeManager REST client