pipeline.PipelineFile
PipelineFile - class
pipeline.objects.variable.PipelineFile(
*
path: str = None,
name: str = None,
remote_id: str = None,
local_id: str = None,
)
Description
The PipelineFile
class allows for local files to be used passed into a graph via a node in the Pipeline
context manager. It also allows for local files to be wrapped with the graph when sent for remote execution.
When a pipeline graph is serialised for remote execution the local file referenced by the path
variable is serialised and sent as well. When the graph is reformed remotely the path
variable will change to the path on the new system.
Parameters
path
(str
) - the path to the local file.name
(str
, optional) - a separate name associated with the file (not affiliated with it's path).remote_id
(str
, optional) - an ID referring to the remote identifier for the object. See example for more information.local_id
(str
, optional) - an ID used for local referencing of theVariable
in the graph by nodes.
Functions
from_schema(cls, schema: PipelineVariableGet) -> PipelineFile
from_schema(cls, schema: PipelineVariableGet) -> PipelineFile
Description
This class function allows for the creation of a PipelineFile
object from a PipelineVariableGet
schema. This is currently not used when running locally but for remote execution.
Parameters
cls
(PipelineFile
) - the class method variable.schema
(PipelineVariableGet
) - the schema to create the pipeline file from.
Examples
from pipeline import Pipeline, PipelineFile
@pipeline_function
def print_file(file_input: PipelineFile) -> None:
with open(file_input.path, "r") as my_file:
print(my_file.read())
with Pipeline("test") as builder:
...
my_file = PipelineFile(path="some_local_file")
print_file(my_file)
...
Remote PipelineFile
PipelineFile
Downloading a remote file to use locally. The remote file is fetched and stored locally in the files cache.
with Pipeline("pf-test") as builder:
...
pfile = PipelineFile(remote_id="remote_file_id")
builder.add_variables(pfile)
...
pf_test = Pipeline.get_pipeline("pf-test")
pcloud.download_remotes(pf_test) # Error is thrown if file is not downloaded
pf_test.run(...)
Using a remote file in when uploading a new pipeline. In this instance, the remote file is not downloaded. Instead, the FileGet
schema in the PipelineFileVariableGet
schema is populated with the fetched file schema using the file id in the remote_id="remote_file_id"
kwarg.
with Pipeline("pf-test") as builder:
...
pfile = PipelineFile(remote_id="remote_file_id")
builder.add_variables(pfile)
...
pf_test = Pipeline.get_pipeline("pf-test")
pcloud.upload_pipeline(pf_test)
Updated 11 months ago