Job Reference

Jobs submitted to DeepSquare use the JSON format. The top level object is the Job.

Job (top level object)

A Job is a finite sequence of instructions.

Field	Type	Description
resources	JobResources!	Allocated resources for the job. Each resource is available as environment variables: $NTASKS: number of allowed parallel tasks $CPUS_PER_TASK: number of CPUs per task $MEM_PER_CPU: MB of memory per CPU $GPUS: total number of GPUS $CPUS: total number of CPUS $MEM: total number of memory in MB Go name: "Resources".
env	[EnvVar!]	Environment variables accessible for the entire job. Go name: "Env".
enableLogging	Boolean	EnableLogging enables the DeepSquare Grid Logger. Go name: "EnableLogging".
input	TransportData	Pull data at the start of the job. It is recommended to set the mode of the data by filling the `inputMode` field. Go name: "Input".
inputMode	Int	InputMode takes an integer that will be used to change the mode recursively (chmod -R) of the input data. The number shouldn't be in octal but in decimal. A mode over 512 is not accepted. Common modes: 511 (user:rwx group:rwx world:rwx) 493 (user:rwx group:r-x world:r-x) 448 (user:rwx group:--- world:---) If null, the mode won't change and will default to the source. Go name: "InputMode".
steps	[Step!]!	Group of steps that will be run sequentially. Go name: "Steps".
output	TransportData	Push data at the end of the job. Continuous sync/push can be enabled using the `continuousOutputSync` flag. Go name: "Output".
continuousOutputSync	Boolean	ContinuousOutputSync will push data during the whole job. This is useful when it is not desired to lose data when the job is suddenly stopped. ContinousOutputSync is not available with HTTP. Go name: "ContinuousOutputSync".
virtualNetworks	[VirtualNetwork!]	A list of virtual network. Can only be used with network namespaces. Go name: "VirtualNetworks".

Examples

Job (minimal)
resources:
  tasks: 1
  gpus: 0
  cpusPerTask: 1
  memPerCpu: 1024
enableLogging: true
steps:
  - name: hello world
    run:
      command: echo "Hello World"

note

A empty command field (command: "") will executes the default container ENTRYPOINT.

Job (full)
resources:
  tasks: 1
  gpus: 0
  cpusPerTask: 1
  memPerCpu: 1024
enableLogging: true
input:
  http:
    url: https://my-server/my-file
inputMode: 493
output:
  http:
    url: https://transfer.sh
env:
  - key: MY_ENV
    value: my_value
continuousOutputSync: true
steps:
  - command: ping 10.0.0.1
    resources:
      tasks: 1
      cpusPerTask: 1
      memPerCpu: 500
    network: slirp4netns # or "pasta" # or "pasta"
    dns: 1.1.1.1
    container:
      image: library/ubuntu:latest
      registry: registry-1.docker.io
    customNetworkInterfaces:
      - wireguard:
          address:
            - 10.0.0.2/24
          privateKey: '<TO FILL: Client PK>'
          peers:
            - publicKey: '<TO FILL: Serv Pub>'
              preSharedKey: '<TO FILL: SharedKey>'
              allowedIPs:
                - 10.0.0.1/32
              persistentKeepalive: 10
              endpoint: 192.168.0.0:51820
    env:
      - key: MY_VAR
        value: myvalue
    workDir: '/app'
    disableCpuBinding: false

`.resources` JobResources

JobResources are the allocated resources for a job in a cluster.

Field	Type	Description
tasks	Int!	Number of tasks which are run in parallel. Can be greater or equal to 1. Go name: "Tasks".
cpusPerTask	Int!	Allocated CPUs per task. Can be greater or equal to 1. Go name: "CPUsPerTask".
memPerCpu	Int!	Allocated memory (MB) per task. Can be greater or equal to 1. Go name: "MemPerCPU".
gpus	Int!	Allocated GPUs for the whole job. Tasks can consume the GPUs by setting `GPUsPerTask` at step level. Can be greater or equal to 0. Go name: "GPUs".

Example

JobResources
resources:
  tasks: 1
  cpusPerTask: 1
  memPerCpu: 1000
  gpusPerTask: 2

`.env[]` EnvVar

An environment variable.

Accessible via: "$key". "Key" name must follows the POSIX specifications (alphanumeric with underscore).

Field	Type	Description
key	String!	Key of the environment variable. Go name: "Key".
value	String!	Value of the environment variable. Go name: "Value".

Field

Type

Description

key

String!

Key of the environment variable.

Go name: "Key".

value

String!

Value of the environment variable.

Go name: "Value".

Example

EnvVar
env:
  - key: MY_ENV
    value: my value

`.virtualNetworks[]` VirtualNetwork

A virtual network is a network that can be used to connect network namespaces.

For now, the virtual network use

Field	Type	Description
name	String!	Name of the virtual network. Use this name to reference the network. Go name: "Name".
gatewayAddress	String!	Gateway address (CIDR). Note this does not forward to the internet. This is only used for NAT traversal. Example: "10.0.0.1/24". IPv6 is also supported. Go name: "GatewayAddress".

Field

Type

Description

name

String!

Name of the virtual network.

Use this name to reference the network.

Go name: "Name".

gatewayAddress

String!

Gateway address (CIDR). Note this does not forward to the internet. This is only used for NAT traversal.

Example: "10.0.0.1/24". IPv6 is also supported.

Go name: "GatewayAddress".

Examples

Virtual Network
virtualNetworks:
  - name: my-network
    gatewayAddress: 10.0.0.1/24

steps:
  - name: do-async
    launch:
      steps:
        - name: server
          run:
            network: slirp4netns
            mapUid: 0
            mapGid: 0
            customNetworkInterfaces:
              - vnet:
                  name: my-network
                  address: 10.0.0.2/24
            command: |-
              nc -l -p 12345

  - name: client
    run:
      network: slirp4netns
      mapUid: 0
      mapGid: 0
      customNetworkInterfaces:
        - vnet:
            name: my-network
            address: 10.0.0.3/24
      command: |-
        while true; do
            echo "Trying to connect..."
            if echo "hello world" | nc 10.0.0.2 12345; then
                echo "Connection successful"
                break
            else
                echo "Connection failed. Retrying in 5 seconds..."
                sleep 5
            fi
        done
        echo "Success"

`.input` and `.output` TransportData

Field	Type	Description
http	HTTPData	Use http to download a file or archive, which will be autoextracted. Go name: "HTTP".
s3	S3Data	Use s3 to sync a file or directory. Go name: "S3".

Field

Type

Description

http

HTTPData

Use http to download a file or archive, which will be autoextracted.

Go name: "HTTP".

S3Data

Use s3 to sync a file or directory.

Go name: "S3".

Examples

TransportData (http)
input:
  http:
    url: https://my-server/my-file

TransportData (s3)
input:
  s3:
    region: us‑east‑2
    bucketUrl: s3://my-bucket
    path: '/'
    accessKeyId: accessKeyId
    secretAccessKey: secretAccessKey
    endpointUrl: https://s3.us‑east‑2.amazonaws.com
    deleteSync: true

`.input.http` and `.output.http` HTTPData

HTTPData describes the necessary variables to connect to a HTTP storage.

Field	Type	Description
url	String!	HTTP or HTTPS URL to a file. Go name: "URL".

Field

Type

Description

url

String!

HTTP or HTTPS URL to a file.

Go name: "URL".

Example

HTTPData
http:
  url: https://my-server/my-file

`.input.s3` and `.output.s3` S3Data

S3Data describes the necessary variables to connect to a S3 storage.

Field	Type	Description
region	String!	S3 region. Example: "us‑east‑2". Go name: "Region".
bucketUrl	String!	The S3 Bucket URL. Must not end with "/". Example: "s3://my-bucket". Go name: "BucketURL".
path	String!	The absolute path to a directory/file inside the bucket. Must start with "/". Go name: "Path".
accessKeyId	String!	An access key ID for the S3 endpoint. Go name: "AccessKeyID".
secretAccessKey	String!	A secret access key for the S3 endpoint. Go name: "SecretAccessKey".
endpointUrl	String!	A S3 Endpoint URL used for authentication. Example: https://s3.us‑east‑2.amazonaws.com Go name: "EndpointURL".
deleteSync	Boolean	DeleteSync removes destination files that doesn't correspond to the source. This applies to any type of source to any type of destination (s3 or filesystem). See: s5cmd sync --delete. If null, defaults to false. Go name: "DeleteSync".

Examples

S3Data
s3:
  region: us‑east‑2
  bucketUrl: s3://my-bucket
  path: '/'
  accessKeyId: accessKeyId
  secretAccessKey: secretAccessKey
  endpointUrl: https://s3.us‑east‑2.amazonaws.com

S3Data (remove non original data)
s3:
  region: us‑east‑2
  bucketUrl: s3://my-bucket
  path: '/'
  accessKeyId: accessKeyId
  secretAccessKey: secretAccessKey
  endpointUrl: https://s3.us‑east‑2.amazonaws.com
  deleteSync: true

`.steps[]` Step

Step is one instruction.

Field	Type	Description
name	String	Name of the instruction. Is used for debugging. Go name: "Name".
dependsOn	[String!]	Depends on wait for async tasks to end before launching this step. DependsOn uses the `handleName` property of a `StepAsyncLaunch`. Only steps at the same level can be awaited. BE WARNED: Uncontrolled `dependsOn` may results in dead locks. Go name: "DependsOn".
if	String	"If" is a boolean test that skips the step if the test is false. The test format is bash and variables such as $PATH or $(pwd) can be expanded. Note that "If" will be run after the "DependsOn". Example: '3 -eq 3 && "${TEST}" = "test"'. Go name: "If".
steps	[Step!]	Group of steps that will be run sequentially. Is exclusive with "for", "launch", "use", "run". Go name: "Steps".
run	StepRun	Run a command if not null. Is exclusive with "for", "launch", "use", "steps". Go name: "Run".
for	StepFor	Run a for loop if not null. Is exclusive with "run", "launch", "use", "steps". Go name: "For".
launch	StepAsyncLaunch	Launch a background process to run a group of commands if not null. Is exclusive with "run", "for", "use", "steps". Go name: "Launch".
use	StepUse	Use a third-party group of steps. Is exclusive with "run", "for", "launch", "steps". Go name: "Use".
catch	[Step!]	Group of steps that will be run sequentially on error. Go name: "Catch".
finally	[Step!]	Group of steps that will be run sequentially after the group of steps or command finishes. Go name: "Finally".

Examples

Step (run)
steps:
  - name: print hello world
    run:
      command: echo 'hello world'

Step (for)
steps:
  - name: print 1 2 3
    for:
      range:
        begin: 1
        end: 3
      steps:
        - name: print variable
          run:
            command: echo "$index"

Step (if)
steps:
  - name: if example
    if: '-f /tmp/lock'
    run:
      command: echo 'show only if /tmp/lock exists'

Step (try-catch)
steps:
  - name: catch block
    catch:
      - name: run only on non-zero error code
        run:
          command: echo $DEEPSQUARE_ERROR_CODE
    run:
      command: exit 1

Step (defer)
steps:
  - name: defer block
    finally:
      - name: run at the end of the scope
    steps:
      - name: run
        run:
          command: echo 'do'

Step (try-catch-finally)
steps:
  - name: catch-finally block
    catch:
      - name: run only on non-zero error code
        run:
          command: echo $DEEPSQUARE_ERROR_CODE
    finally:
      - name: run anyway
    steps:
      - name: run
        run:
          command: exit 1

`.steps[].run` StepRun

StepRun is one script executed with the shell.

A temporary shared storage is accessible through the $STORAGE_PATH environment variable.

Availables caches can be used by invoking one of the following environment variable:

Environment variables	Lifecycle
STORAGE_PATH	job duration
DEEPSQUARE_TMP or DEEPSQUARE_SHARED_TMP	provider's policy
DEEPSQUARE_SHARED_WORLD_TMP	provider's policy
DEEPSQUARE_DISK_TMP	node reboot or provider's policy
DEEPSQUARE_DISK_WORLD_TMP	node reboot or provider's policy

echo "KEY=value" >> "$DEEPSQUARE_ENV" can be used to share environment variables between steps.

$DEEPSQUARE_INPUT is the path that contains imported files.

$DEEPSQUARE_OUTPUT is the staging directory for uploading files.

Field	Type	Description
command	String!	Command specifies a shell script or CMD. If container is used, `command` and `shell` automatically overwrite the ENTRYPOINT and CMD. If you want to execute the default ENTRYPOINT and CMD, set the `command` empty. If you want to execute the default ENTRYPOINT with a custom CMD, set the `command` to your args and set the `shell` to `ENTRYPOINT`. Right now, ENTRYPOINT cannot be overriden for security reasons. Go name: "Command".
shell	String	Shell to use. Use "ENTRYPOINT" to use the default ENTRYPOINT. Accepted: /bin/bash, /bin/ash, /bin/sh, ENTRYPOINT Default: /bin/sh Go name: "Shell".
resources	StepRunResources	Allocated resources for the command. Go name: "Resources".
container	ContainerRun	Container definition. If null, run on the host. Go name: "Container".
network	String	Type of core networking functionality. Either: "host" (default) or "slirp4netns" (rootless network namespace) or "pasta" (simple rootless network namespace) "slirp4netns" uses "slirp" to forward traffic from a network namespace to the host. "pasta" is an alternative to "slirp4netns" and uses "passt" to forward traffic from a network namespace to the host. Go name: "Network".
dns	[String!]	Configuration for the DNS in "slirp4netns" or "pasta" mode. ONLY enabled if network is "slirp4netns" or "pasta". A comma-separated list of DNS IP. Go name: "DNS".
customNetworkInterfaces	[NetworkInterface!]	Add custom network interfaces. ONLY enabled if network is "slirp4netns" or "pasta". You may need to map to root to be able to create network interfaces like Wireguard. The default network interface is net0, which is a TAP interface connecting the host and the network namespace. Go name: "CustomNetworkInterfaces".
env	[EnvVar!]	Environment variables accessible over the command. Go name: "Env".
mapUid	Int	Remap UID. Does not grant elevated system permissions, despite appearances. MapUID doesn't work very well with Apptainer. You can still map to root, but you cannot map to an unknown user. Go name: "MapUID".
mapGid	Int	Remap GID. Does not grant elevated system permissions, despite appearances. Go name: "MapGID".
workDir	String	Working directory. If the "default" (Enroot) container runtime is used, it will use the `--container-workdir` flag. If the "apptainer" container runtime is used, the `--pwd` flag will be passed. If no container runtime is used, `cd` will be executed first. If null, default to use $STORAGE_PATH as working directory. Go name: "WorkDir".
disableCpuBinding	Boolean	DisableCPUBinding disables process affinity binding to tasks. Can be useful when running MPI jobs. If null, defaults to false. Go name: "DisableCPUBinding".
mpi	String	MPI selection. Must be one of: none, pmix_v4, pmi2. If null, will default to infrastructure provider settings (which may not be what you want). Go name: "Mpi".

Examples

StepRun (minimal)
run:
  command: echo 'hello world'

StepRun (with resource limitation)
run:
  command: echo 'hello world'
  resources:
    tasks: 1
    cpusPerTask: 1
    memPerCpu: 500

StepRun (with container)
run:
  command: echo 'hello world'
  container:
    image: library/ubuntu:latest
    registry: registry-1.docker.io

StepRun (with rootless network namespace)
run:
  command: echo 'hello world'
  network: slirp4netns # or "pasta"

StepRun (with a Wireguard tunnel)
run:
  command: ping 10.0.0.1
  network: slirp4netns # or "pasta"
  dns: 1.1.1.1
  customNetworkInterfaces:
    - wireguard:
        address:
          - 10.0.0.2/24
        privateKey: '<TO FILL: Client PK>'
        peers:
          - publicKey: '<TO FILL: Serv Pub>'
            preSharedKey: '<TO FILL: SharedKey>'
            allowedIPs:
              - 10.0.0.1/32
            persistentKeepalive: 10
            endpoint: 192.168.0.0:51820

StepRun (full)
run:
  command: ping 10.0.0.1
  resources:
    tasks: 1
    cpusPerTask: 1
    memPerCpu: 500
  network: slirp4netns # or "pasta"
  dns: 1.1.1.1
  container:
    image: library/ubuntu:latest
    registry: registry-1.docker.io
  customNetworkInterfaces:
    - wireguard:
        address:
          - 10.0.0.2/24
        privateKey: '<TO FILL: Client PK>'
        peers:
          - publicKey: '<TO FILL: Serv Pub>'
            preSharedKey: '<TO FILL: SharedKey>'
            allowedIPs:
              - 10.0.0.1/32
            persistentKeepalive: 10
            endpoint: 192.168.0.0:51820
  env:
    - key: MY_VAR
      value: myvalue
  workDir: '/app'
  disableCpuBinding: false

`.steps[].run.resources` StepRunResources

StepRunResources are the allocated resources for a command in a job.

Field	Type	Description
tasks	Int	Number of tasks which are run in parallel. Can be greater or equal to 1. If null, default to 1. Go name: "Tasks".
cpusPerTask	Int	Allocated CPUs per task. Can be greater or equal to 1. If null, defaults to the job resources. Go name: "CPUsPerTask".
memPerCpu	Int	Allocated memory (MB) per task. Can be greater or equal to 1. If null, defaults to the job resources. Go name: "MemPerCPU".
gpusPerTask	Int	Allocated GPUs per task. Can be greater or equal to 0. If null, defaults to 0. Go name: "GPUsPerTask".

Examples

StepRunResources (partial)
run:
  resources:
    tasks: 1
    # gpusPerTask: 0
    # cpusPerTask: inherit from job
    # memPerCpu: inherit from job

StepRunResources (full)
run:
  resources:
    tasks: 1
    cpusPerTask: 1
    memPerCpu: 1000
    gpusPerTask: 0

`.steps[].run.container` ContainerRun

Field	Type	Description
image	String!	Run the command inside a container with Enroot. Format: image:tag. Registry and authentication is not allowed on this field. If the default container runtime is used: Use an absolute path to load a squashfs file. By default, it will search inside $STORAGE_PATH. /input will be equivalent to $DEEPSQUARE_INPUT, /output is $DEEPSQUARE_OUTPUT If apptainer=true: Use an absolute path to load a sif file or a squashfs file. By default, it will search inside $STORAGE_PATH. /input will be equivalent to $DEEPSQUARE_INPUT, /output is $DEEPSQUARE_OUTPUT Examples: library/ubuntu:latest /my.squashfs Go name: "Image".
mounts	[Mount!]	[DEPRECATED] Mounts decribes a Bind Mount. Please use predefined mounts like $STORAGE_PATH, $DEEPSQUARE_TMP, ... Go name: "Mounts".
username	String	Username of a basic authentication. Go name: "Username".
password	String	Password of a basic authentication. Go name: "Password".
registry	String	Container registry host. Defaults to registry-1.docker.io. Go name: "Registry".
apptainer	Boolean	Run with Apptainer as Container runtime instead of Enroot. By running with apptainer, you get access Deepsquare-hosted images. When running Apptainer, the container file system is read-only. Defaults to false. Go name: "Apptainer".
deepsquareHosted	Boolean	Use DeepSquare-hosted images. By setting to true, apptainer will be set to true. Go name: "DeepsquareHosted".
x11	Boolean	X11 mounts /tmp/.X11-unix in the container. Go name: "X11".
mountHome	Boolean	Mount the home directories. Go name: "MountHome".
readOnlyRootFS	Boolean	Disable write permissions on the container root file system. Does not applies to mounts. Go name: "ReadOnlyRootFS".

Example

ContainerRun (public registry)
container:
  image: library/ubuntu:latest
  registry: registry-1.docker.io

ContainerRun (private registry)
container:
  image: library/ubuntu:latest
  registry: registry-1.docker.io
  username: john
  password: password

ContainerRun (Apptainer runtime, public registry)
container:
  image: library/ubuntu:latest
  registry: registry-1.docker.io
  apptainer: true

ContainerRun (Apptainer runtime, Deepsquare-Hosted images)
container:
  image: library/stable-diffusion:latest
  registry: registry-1.deepsquare.run
  apptainer: true
  deepsquareHosted: true

ContainerRun (x11 mount)
container:
  image: library/stable-diffusion:latest
  registry: registry-1.deepsquare.run
  x11: true

ContainerRun (full example)
container:
  image: ubuntu:latest
  registry: registry-1.docker.io
  mounts:
    - hostDir: '/host'
      containerDir: '/container'
      options: ro
  username: john
  password: password
  apptainer: true
  deepsquareHosted: false
  x11: false

`.steps[].run.container.mounts[]`Mount

DEPRECATED: Mount decribes a Bind Mount.

Mount is now deprecated. Please use predefined mounts like $STORAGE_PATH, $DEEPSQUARE_TMP, ...

Field	Type	Description
hostDir	String!	Directory on the host to be mounted inside the container. Go name: "HostDir".
containerDir	String!	Target directory inside the container. Go name: "ContainerDir".
options	String!	Options modifies the mount options. Accepted: ro, rw Go name: "Options".

Field

Type

Description

hostDir

String!

Directory on the host to be mounted inside the container.

Go name: "HostDir".

containerDir

String!

Target directory inside the container.

Go name: "ContainerDir".

options

String!

Options modifies the mount options.

Accepted: ro, rw

Go name: "Options".

Example

Mount
mounts:
  - hostDir: '/host'
    containerDir: '/container'
    options: ro

Mount-bind tmp (can be used to pass data between steps)
mounts:
  hostDir: '/tmp'
  containerDir: '/tmp'
  options: rw

`.steps[].run.customNetworkInterfaces[]`NetworkInterface

Connect a network interface on a StepRun.

The network interface is connected via slirp4netns.

If using wireguard, please mapUid to root (mapUid=0).

Field	Type	Description
wireguard	Wireguard	Use the wireguard transport. Go name: "Wireguard".
bore	Bore	Use the bore transport. Go name: "Bore".
vnet	VNet	Use a DeepSquare-managed virtual network for inter-step communication. It uses Wireguard to interconnect the steps. The communication are encrypted. Go name: "VNet".

Field

Type

Description

wireguard

Wireguard

Use the wireguard transport.

Go name: "Wireguard".

bore

Bore

Use the bore transport.

Go name: "Bore".

vnet

VNet

Use a DeepSquare-managed virtual network for inter-step communication.

It uses Wireguard to interconnect the steps. The communication are encrypted.

Go name: "VNet".

Example

NetworkInterface (wireguard)
customNetworkInterfaces:
  - wireguard:
    address:
      - 10.0.0.2/24
    privateKey: '<TO FILL: Client PK>'
    peers:
      - publicKey: '<TO FILL: Serv Pub>'
        preSharedKey: '<TO FILL: SharedKey>'
        allowedIPs:
          - 10.0.0.1/32
        persistentKeepalive: 10
        endpoint: 192.168.0.0:51820

You can generate a keypair and a shared key with:

/bin/sh
wg genkey | tee privatekey | wg pubkey > pubkey
wg genkey > sharedkey

`.steps[].run.customNetworkInterfaces[].wireguard` Wireguard

Wireguard VPN Transport for StepRun.

The Wireguard VPN can be used as a gateway for the steps. All that is needed is a Wireguard server outside the cluster that acts as a public gateway.

The interfaces are named wg0, wg1, ..., wgN.

Wireguard transport uses UDP hole punching to connect to the VPN Server.

Disabled settings: PreUp, PostUp, PreDown, PostDown, ListenPort, Table, MTU, SaveConfig.

If these features are necessary, please do contact DeepSquare developpers!

Field	Type	Description
address	[String!]	The IP addresses of the wireguard interface. Format is a CIDRv4 (X.X.X.X/X) or CIDRv6. Recommendation is to take one IP from the 10.0.0.0/24 range (example: 10.0.0.2/24). Go name: "Address".
privateKey	String!	The client private key. Go name: "PrivateKey".
peers	[WireguardPeer!]	The peers connected to the wireguard interface. Go name: "Peers".

Field

Type

Description

address

[String!]

The IP addresses of the wireguard interface.

Format is a CIDRv4 (X.X.X.X/X) or CIDRv6.

Recommendation is to take one IP from the 10.0.0.0/24 range (example: 10.0.0.2/24).

Go name: "Address".

privateKey

String!

The client private key.

Go name: "PrivateKey".

peers

[WireguardPeer!]

The peers connected to the wireguard interface.

Go name: "Peers".

Example

Wireguard
wireguard:
  address:
    - 10.0.0.2/24
  privateKey: '<TO FILL: Client PK>'
  peers:
    - publicKey: '<TO FILL: Serv Pub>'
      preSharedKey: '<TO FILL: SharedKey>'
      allowedIPs:
        - 10.0.0.1/32
      persistentKeepalive: 10
      endpoint: 192.168.0.0:51820

You can generate a keypair and a shared key with:

/bin/sh
wg genkey | tee privatekey | wg pubkey > pubkey
wg genkey > sharedkey

`.steps[].run.customNetworkInterfaces[].wireguard.peers[]` WireguardPeer

A Wireguard Peer.

Field	Type	Description
publicKey	String!	The peer private key. Go name: "PublicKey".
preSharedKey	String	The peer pre-shared key. Go name: "PreSharedKey".
allowedIPs	[String!]	Configuration of wireguard routes. Format is a CIDRv4 (X.X.X.X/X) or CIDRv6. 0.0.0.0/0 (or ::/0) would forward all packets to the tunnel. If you plan to use the Wireguard VPN as a gateway, you MUST set this IP range. <server internal IP>/32 (not the server's public IP) would forward all packets to the tunnel with the server IP as the destination. MUST be set. <VPN IP range> would forward all packets to the tunnel with the local network as the destination. Useful if you want peers to communicate with each other and want the gateway to act as a router. Go name: "AllowedIPs".
endpoint	String	The peer endpoint. Format is IP:port. This would be the Wireguard server. Go name: "Endpoint".
persistentKeepalive	Int	Initiate the handshake and re-initiate regularly. Takes seconds as parameter. 25 seconds is recommended. You MUST set the persistent keepalive to enables UDP hole-punching. Go name: "PersistentKeepalive".

Example

WireguardPeer
peers:
  - publicKey: '<TO FILL: Serv Pub>'
    preSharedKey: '<TO FILL: SharedKey>'
    allowedIPs:
      - 10.0.0.1/32
    persistentKeepalive: 10
    endpoint: 192.168.0.0:51820

You can generate a keypair and a shared key with:

/bin/sh
wg genkey | tee privatekey | wg pubkey > pubkey
wg genkey > sharedkey

`.steps[].run.customNetworkInterfaces[].bore` Bore

jkuri/bore tunnel Transport for StepRun.

Bore is a proxy to expose TCP sockets.

Field	Type	Description
address ⚠️	String	Bore server IP/Address. Deprecated: Use boreAddress. Go name: "Address". ⚠️ DEPRECATED address is deprecated. Use boreAddress instead.
boreAddress	String	Bore server IP/Address:Port. Go name: "BoreAddress".
port ⚠️	Int	The bore server port. Deprecated: Use boreAddress. Go name: "Port". ⚠️ DEPRECATED port is deprecated. Use boreAddress instead.
targetPort	Int!	Target port. Go name: "TargetPort".
secret	String	Secret used to authenticate on a Bore server. This secret is used to identify the client. Go name: "Secret".

Example

Bore
bore:
  boreAddress: bore.deepsquare.run:2200
  targetPort: 8080
  secret: <token>

`steps[].run.customNetworkInterfaces[].vnet` VNet

Use VNet as network interface.

Field	Type	Description
name	String!	Name of the network to be used. Must exists. See Job.Networks. Go name: "Name".
address	String!	Address (CIDR) of the interface. Example: "10.0.0.2/24" which means The interface's IP is 10.0.0.2. Route packets with destination 10.0.0.0/24 to that interface. Go name: "Address"

Field

Type

Description

name

String!

Name of the network to be used. Must exists.

See Job.Networks.

Go name: "Name".

address

String!

Address (CIDR) of the interface.

Example: "10.0.0.2/24" which means

The interface's IP is 10.0.0.2.
Route packets with destination 10.0.0.0/24 to that interface.

Go name: "Address"

`.steps[].for` StepFor

StepFor describes a for loop.

Field	Type	Description
parallel	Boolean!	Do a parallel for loop. Each iteration is run in parallel. Go name: "Parallel".
items	[String!]	Item accessible via the {{ .Item }} variable. Index accessible via the $item variable. Exclusive with "range". Go name: "Items".
range	ForRange	Index accessible via the $index variable. Exclusive with "items". Go name: "Range".
steps	[Step!]!	Steps are run sequentially in one iteration. Go name: "Steps".

Example

StepFor (items)
for:
  parallel: true
  items: [a, b, c]
  steps:
    - name: print variable
      run:
        command: echo "$item"

Will prints "a", "b", "c".

StepFor (range)
for:
  parallel: true
  range:
    begin: 1
    end: 3
  steps:
    - name: print variable
      run:
        command: echo "$index"

Will prints "1", "2", "3".

`.steps[].for.range` ForRange

ForRange describes the parameter for a range loop.

Field	Type	Description
begin	Int!	Begin is inclusive. Go name: "Begin".
end	Int!	End is inclusive. Go name: "End".
increment	Int	Increment counter by x count. If null, defaults to 1. Go name: "Increment".

Field

Type

Description

begin

Int!

Begin is inclusive.

Go name: "Begin".

end

Int!

End is inclusive.

Go name: "End".

increment

Int

Increment counter by x count. If null, defaults to 1.

Go name: "Increment".

Example

ForRange
range:
  begin: 1
  end: 3

Will results in 3 steps.

`.steps[].launch` StepAsyncLaunch

StepAsyncLaunch describes launching a background process.

StepAsyncLaunch will be awaited at the end of the job.

Field	Type	Description
handleName	String	HandleName is the name used to await (dependsOn field of the Step). Naming style is snake_case. Case is insensitive. No symbol allowed. Go name: "HandleName".
signalOnParentStepExit	Int	SignalOnParentStepExit sends a signal to the step and sub-steps when the parent step ends. This function can be used as a cleanup function to avoid a zombie process. Zombie processes will continue to run after the main process dies and therefore will not stop the job. If null, SIGTERM will be sent. If 0, no signal will be sent. Current signal : 1 SIGHUP Hang-up detected on the control terminal or death of the control process. 2 SIGINT Abort from keyboard 3 SIGQUIT Quit the keyboard 9 SIGKILL If a process receives this signal, it must quit immediately and will not perform any cleaning operations. 15 SIGTERM Software stop signal It is STRONGLY RECOMMENDED to use SIGTERM to gracefully exit a process. SIGKILL is the most abrupt and will certainly work. If no signal is sent, the asynchronous step will be considered a fire and forget asynchronous step and will have to terminate itself to stop the job. WARNING: the "no signal sent" option is subject to removal to avoid undefined behavior. Please refrain from using it. Go name: "SignalOnParentStepExit".
steps	[Step!]!	Steps are run sequentially. Go name: "Steps".

Field

Type

Description

handleName

String

HandleName is the name used to await (dependsOn field of the Step).

Naming style is snake_case. Case is insensitive. No symbol allowed.

Go name: "HandleName".

signalOnParentStepExit

Int

SignalOnParentStepExit sends a signal to the step and sub-steps when the parent step ends.

This function can be used as a cleanup function to avoid a zombie process.

Zombie processes will continue to run after the main process dies and therefore will not stop the job.

If null, SIGTERM will be sent. If 0, no signal will be sent.

Current signal :

1 SIGHUP Hang-up detected on the control terminal or death of the control process. 2 SIGINT Abort from keyboard 3 SIGQUIT Quit the keyboard 9 SIGKILL If a process receives this signal, it must quit immediately and will not perform any cleaning operations. 15 SIGTERM Software stop signal

It is STRONGLY RECOMMENDED to use SIGTERM to gracefully exit a process. SIGKILL is the most abrupt and will certainly work.

If no signal is sent, the asynchronous step will be considered a fire and forget asynchronous step and will have to terminate itself to stop the job.

WARNING: the "no signal sent" option is subject to removal to avoid undefined behavior. Please refrain from using it.

Go name: "SignalOnParentStepExit".

steps

[Step!]!

Steps are run sequentially.

Go name: "Steps".

Examples

Async-await
resources:
  tasks: 2
  cpusPerTask: 4
  memPerCpu: 4096
  gpus: 0

steps:
  ## Launch async
  - launch:
      handleName: task1
      steps:
        - name: work
          run:
            command: echo "Working"; sleep 15; echo "Working done"
  - name: do-stuff-in-foreground
    run:
      command: echo I work fast
  - name: wait for task1
    ## Await for task1
    dependsOn:
      - task1
    run:
      command: echo Task 1 is done!

Fire-and-forget (explicit safe)
resources:
  tasks: 2
  cpusPerTask: 4
  memPerCpu: 4096
  gpus: 0

steps:
  ## Launch async
  - launch:
      ## If the parent dies, async tasks will receive the "15" (SIGTERM) signal
      signalOnParentStepExit: 15
      steps:
        - name: work
          run:
            command: echo "Working"; sleep 15; echo "Working done"

Fire-and-forget (explicit unsafe)
resources:
  tasks: 2
  cpusPerTask: 4
  memPerCpu: 4096
  gpus: 0

steps:
  ## Launch async
  - launch:
      ## If the parent dies, async tasks will receive no signal
      signalOnParentStepExit: 0
      steps:
        - name: work
          run:
            command: echo "Working"; sleep 15; echo "Working done"

`.steps[].use` StepUse

Field	Type	Description
source	String!	Source of the group of steps. Syntax: <url>@<tag/hash> Example: github.com/example/my-module@v1 Example: github.com/example/module-monorepo/my-module@v1 The host must be a git repository accessible via HTTPS. The path must indicates a directory. For example, `/my-module` indicates the root directory of the repository `my-module`. `module-monorepo/my-module` indicates the subdirectory `my-module` of the repository `module-monorepo`. Go name: "Source".
args	[EnvVar!]	Arguments to be passed as inputs to the group of steps. Go name: "Args".
exportEnvAs	String	Environment variables exported with be prefixed with the value of this field. Exemple: If exportEnvAs=MY_MODULE, and KEY is exported. Then you can invoke ${MY_MODULE_KEY} environment variable. Go name: "ExportEnvAs".
steps	[Step!]	Additional children steps to the module. If the module allow children steps, these steps will be passed to the module to replace {{ .Step.Run.Steps }}. Go name: "Steps".

Example

StepUse
steps:
  - name: use external module
    use:
      source: github.com/deepsquare-io/workflow-module-example@5ca6163
      args:
        - key: WHO
          value: me
      exportEnvAs: HELLO_WORLD
  - name: repeat
    run:
      command: echo ${HELLO_WORLD_RESULT}

Scalars

Boolean

The Boolean scalar type represents true or false.

Int

The Int scalar type represents non-fractional signed whole numeric values. Int can represent values between -(2^31) and 2^31 - 1.

String

The String scalar type represents textual data, represented as UTF-8 character sequences. The String type is most often used to represent free-form human-readable text.

Job Reference

Job (top level object)​

.resources JobResources​

.env[] EnvVar​

.virtualNetworks[] VirtualNetwork​

.input and .output TransportData​

.input.http and .output.http HTTPData​

.input.s3 and .output.s3 S3Data​

.steps[] Step​

.steps[].run StepRun​

.steps[].run.resources StepRunResources​

.steps[].run.container ContainerRun​

.steps[].run.container.mounts[]Mount​

.steps[].run.customNetworkInterfaces[]NetworkInterface​

.steps[].run.customNetworkInterfaces[].wireguard Wireguard​

.steps[].run.customNetworkInterfaces[].wireguard.peers[] WireguardPeer​

.steps[].run.customNetworkInterfaces[].bore Bore​

steps[].run.customNetworkInterfaces[].vnet VNet​

.steps[].for StepFor​

.steps[].for.range ForRange​

.steps[].launch StepAsyncLaunch​

.steps[].use StepUse​

Scalars​

Boolean​

Int​

String​

Job (top level object)

`.resources` JobResources

`.env[]` EnvVar

`.virtualNetworks[]` VirtualNetwork

`.input` and `.output` TransportData

`.input.http` and `.output.http` HTTPData

`.input.s3` and `.output.s3` S3Data

`.steps[]` Step

`.steps[].run` StepRun

`.steps[].run.resources` StepRunResources

`.steps[].run.container` ContainerRun

`.steps[].run.container.mounts[]`Mount

`.steps[].run.customNetworkInterfaces[]`NetworkInterface

`.steps[].run.customNetworkInterfaces[].wireguard` Wireguard

`.steps[].run.customNetworkInterfaces[].wireguard.peers[]` WireguardPeer

`.steps[].run.customNetworkInterfaces[].bore` Bore

`steps[].run.customNetworkInterfaces[].vnet` VNet

`.steps[].for` StepFor

`.steps[].for.range` ForRange

`.steps[].launch` StepAsyncLaunch

`.steps[].use` StepUse

Scalars

Boolean

Int

String