Pipeline - class

Source on GitHub

pipeline.objects.pipeline.Pipeline(
    new_pipeline_name: str,
  compute_type: str = "gpu",
  min_gpu_vram_mb: int = None,
)

Description

The Pipeline class is used to either start a new context manager session to create a graph, or to interact with graph objects currently stored in Pipeline. As it acts as both a context manager and object store there are various static and non static functions used in both situations.

When the context manager is closed, the resultant computational graph is stored inside of the Pipeline object and must be retrieved.

Parameters

• new_pipeline_name (str) - the name of the pipeline graph to be created, this is used to obtain the graph later.
• compute_type (str, optional) - the compute resource required for this pipeline, either cpu or gpu. This is only used when running the pipeline on an external service.
• min_gpu_vram_mb (int, optional) - the minimum required amount of GPU memory required to run the pipeline. This is only used when running the pipeline on an external service.

Functions

output(self, *outputs: Variable) -> None

Description

Adds Variable objects to the output list for the pipeline. When the pipeline graph finishes executing via the run call, the real value of these Variable objects that have been computed from the graph are returned.

Parameters

• *outputs (Variable) - an arbitrary number of Variable objects.

get_pipeline(graph_name: str) -> Graph

Description

Retrieves a pipeline graph based on it's name. Typically this will be used shortly after the context manager.

Parameters

• graph_name (str) - the name of the pipeline.

add_variable(variable: Variable) -> None

Description

Adds a variable to the graph being constructed inside of the context manager. this function cannot be called outside of the context manager.

Parameters

• variable (Variable) - the variable to be added.

add_variables(*variables: Variable) -> None

Description

The same as add_variable but accepts an arbitrary number of inputs.

Parameters

• *variables (Variable) - an arbitrary list of Variable objects to be added.

add_function(function: Function) -> None

Description

This should not be used by a user and is called automatically when a function is called in the context manager to add it to the graph.

Parameters

• function (Function) - the function object to be added to the graph.

add_graph_node(graph_node: GraphNode) -> None

Description

This is should not be used by a user and is called automatically when a function is called in the context manager as the input, output, and function are all used to construct the node.

Parameters

• graph_node (GraphNode) - the GraphNode object to be added to the graph.

Examples

Basic usage of Pipeline context manager.

from pipeline import Pipeline, Variable, pipeline_function


@pipeline_function
def add_numbers(a: int, b: int) -> int:
    return a + b


with Pipeline("add-numbers") as pipeline:
    a = Variable(type_class=int, is_input=True)
    b = Variable(type_class=int, is_input=True)
    pipeline.add_variables(a, b)

    c = add_numbers(a, b)

    pipeline.output(c)