Pipeline steps

​

What is pipeline steps?

In VESSL Pipeline, Pipeline steps are the building blocks of a pipeline. Each step defines a specific action or decision that needs to be executed in the pipeline.

There are several types of pipeline steps available:

• Run: Create a container on the cluster and execute a specific code or command.
• Manual Input: Receive input from the user and define variables to use in the pipeline execution.
• Judgement: Request the user to make a manual decision. Used often for human-in-the-loop workflows.
• If: Define a condition and execute a specific step based on the condition.
• Fail: Terminate the execution and mark it as a failure
• Notification: Send a notification to a specific email address.
• Trigger Dispatch: Execute another pipeline in the project.

​

Types of pipeline steps

​

RunStep

RunStep is the most common step in the pipeline. It creates a container on the cluster and executes a specific code or command - such as training a model, running a batch inference, or deploying a model. A RunStep is composed of the following sections:

​

Metadata

Metadata defines the title, key and description of the step.

• Step title: The title of the step.
• Step key: The unique key of the step.
• Description: The description of the step.

​

Resources

Resources defines where and how the container will be executed.

• Cluster: The cluster where the container will be executed.
• Resource: Resource spec of the container - i.e. How much CPU, memory, and GPU resources the container will use.
• Image: The container image that will be used to execute the code.

​

Task

Task defines the command and volume mounts of the container to be executed.

• Volumes: The volume mounts that will be mounted to the container. You can mount the dataset, model, or any other files to the specified path in the container.
• Command: The command that will be executed in the container. You can define the command as a string or an array of strings.
• Ports: The ports that will be exposed in the container. Once you expose the port and the container is running, you can access the container via the exposed port.

​

Variables

Variables defines environment variables that will be injected into the container.

• Environment variables: The environment variables that will be injected into the container.
• Output variables: The output variables that will be generated from the container. You can export the output variables on the code executed in the container using VESSL SDK like below:

  import vessl
  
  vessl.update_context_variables(data: {"SOME_VARIABLE": "SOME_VALUE"})
  

​

ManualInputStep

ManualInputStep is used to receive input from the user and define variables to use in the pipeline execution. It is often used to define hyperparameters, dataset paths, or any other variables that need to be defined manually.

​

Metadata

Metadata defines the title, key and description of the step.

• Step title: The title of the step.
• Step key: The unique key of the step.
• Description: The description of the step.

​

Input Configuration

Input Configuration defines the input variables that will be injected by the user.

• Assignee: The user who will be assigned to define the input variables. It can be reassigned during the pipeline execution.
• Input parameter: The input variables that will be defined by the user. You can define the name, description and default value of the input variable.

​

JudgementStep

JudgementStep is similar to the ManualInputStep, but it is used to request the user to make a binary decision (Yes/No). It has two output edges so that you can define the next steps based on the decision.

​

Metadata

Metadata defines the title, key, description and assignee of the step.

• Step title: The title of the step.
• Step key: The unique key of the step.
• Description: The description of the step.
• Assignee: The user who will be assigned to make the decision.

​

IfStep

IfStep is used to define a condition and control workflow based on the condition. It has two output edges: one for the true condition and the other for the false condition.

​

Metadata

Metadata defines the title, key, description and assignee of the step.

• Step title: The title of the step.
• Step key: The unique key of the step.
• Description: The description of the step.

​

Condition

Condition defines the condition that will be evaluated.

• Source: Source to get the value. You can use Pipeline variable, Step variable or Constant for the source.
• Variable (Value): The variable that will be used in the condition. If the source is Constant, you can define the value directly.
• Operator: Comparison operator. You can use ==, !=, >, <, >=, <= operators.

​

FailStep

FailStep marks the pipeline execution as a failure and terminates the execution. It is often used to handle errors or evaluation failure cases in the workflow.

​

Metadata

Metadata defines the title, key, description and assignee of the step.

• Step title: The title of the step.
• Step key: The unique key of the step.
• Description: The description of the step.

​

NotificationStep

NotificationStep sends a notification to a specific email address. You can define the title, message, and email address of the notification.

​

Metadata

Metadata defines the title, key, description and assignee of the step.

• Step title: The title of the step.
• Step key: The unique key of the step.
• Description: The description of the step.

​

Condition

Condition defines the title, message, and email address of the notification.

• Email address: The email address that will receive the notification.
• Email subject: The subject of the email.
• Email content: The content of the email.

​

TriggerDispatchStep

TriggerDispatchStep works differently from other steps. It does not execute any code or command. Instead, it triggers another pipeline in the project.

​

Metadata

Metadata defines the configuration which pipeline will be triggered.

• Step title: The title of the step.
• Step key: The unique key of the step.
• Description: The description of the step.
• Pipeline to trigger: The pipeline that will be triggered.
• Target revision: The revision of the pipeline that will be triggered.
• Input variables: The input variables that will be injected into the triggered pipeline.