pipeline.PipelineFile

PipelineFile - class

Source on GitHub

pipeline.objects.variable.PipelineFile(
    *
  path: str = None,
  name: str = None,
  remote_id: str = None,
  local_id: str = None,
)

Description

The PipelineFile class allows for local files to be used passed into a graph via a node in the Pipeline context manager. It also allows for local files to be wrapped with the graph when sent for remote execution.

When a pipeline graph is serialised for remote execution the local file referenced by the path variable is serialised and sent as well. When the graph is reformed remotely the path variable will change to the path on the new system.

Parameters

  • path (str) - the path to the local file.
  • name (str, optional) - a separate name associated with the file (not affiliated with it's path).
  • remote_id (str, optional) - an ID referring to the remote identifier for the object. Not used locally yet.
  • local_id (str, optional) - an ID used for local referencing of the Variable in the graph by nodes.

Functions

from_schema(cls, schema: PipelineVariableGet) -> PipelineFile

Description

This class function allows for the creation of a PipelineFile object from a PipelineVariableGet schema. This is currently not used when running locally but for remote execution.

Parameters

  • cls (PipelineFile) - the class method variable.
  • schema (PipelineVariableGet) - the schema to create the pipeline file from.

Examples

from pipeline import Pipeline, PipelineFile

@pipeline_function
def print_file(file_input: PipelineFile) -> None:
    with open(file_input.path, "r") as my_file:
    print(my_file.read())
    
with Pipeline("test") as builder:
    ...
    
  my_file = PipelineFile(path="some_local_file")
  print_file(my_file)
  
  ...