Compile a Pipeline

To submit a pipeline for execution, you must compile it to YAML with the KFP SDK compiler:

from kfp import dsl
from kfp import compiler

@dsl.component
def comp(message: str) -> str:
    print(message)
    return message

@dsl.pipeline
def my_pipeline(message: str) -> str:
    """My ML pipeline."""
    return comp(message=message).output

compiler.Compiler().compile(my_pipeline, package_path='pipeline.yaml')

In this example, the compiler creates a file called pipeline.yaml, which contains a hermetic representation of your pipeline. The output is called intermediate representation (IR) YAML. You can view an example of IR YAML on GitHub. The contents of the file is the serialized PipelineSpec protocol buffer message and is not intended to be human-readable.

You can find human-readable information about the pipeline in the comments at the top of the compiled YAML:

# PIPELINE DEFINITION
# Name: my-pipeline
# Description: My ML pipeline.
# Inputs:
#    message: str
# Outputs:
#    Output: str
...

You can also compile components, as opposed to pipelines, to IR YAML:

@dsl.component
def comp(message: str) -> str:
    print(message)
    return message

compiler.Compiler().compile(comp, package_path='component.yaml')

Compiler arguments

The Compiler.compile method accepts the following arguments:

Type checking

By default, the DSL compiler statically type checks your pipeline to ensure type consistency between components that pass data between one another. Static type checking helps identify component I/O inconsistencies without having to run the pipeline, shortening development iterations.

Specifically, the type checker checks for type equality between the type of data a component input expects and the type of the data provided. See Data Types for more information about KFP data types.

For example, for parameters, a list input may only be passed to parameters with a typing.List annotation. Similarly, a float may only be passed to parameters with a float annotation.

Input data types and annotations must also match for artifacts, with one exception: the Artifact type is compatible with all other artifact types. In this sense, the Artifact type is both the default artifact type and an artifact “any” type.

As described in the following section, you can disable type checking.

IR YAML

The IR YAML is an intermediate representation of a compiled pipeline or component. It is an instance of the PipelineSpec protocol buffer message type, which is a platform-agnostic pipeline representation protocol. It is considered an intermediate representation because the KFP backend compiles PipelineSpec to Argo Workflow YAML as the final pipeline definition for execution.

Unlike the v1 component YAML, the IR YAML is not intended to be written directly.

While IR YAML is not intended to be easily human readable, you can still inspect it if you know a bit about its contents: