Common Workflow Language (CWL) Command Line Tool Description, v1.0.2 §

This version:

• https://w3id.org/cwl/v1.0/

Current version:

• https://w3id.org/cwl/

Authors:

• Peter Amstutz pamstutz@veritasgenetics.com, Arvados Project, Veritas Genetics
• Michael R. Crusoe mrc@commonwl.org, Common Workflow Language project
• Nebojša Tijanić nebojsa.tijanic@sbgenomics.com, Seven Bridges Genomics

Contributors:

• Brad Chapman bchapman@hsph.harvard.edu, Harvard Chan School of Public Health
• John Chilton jmchilton@gmail.com, Galaxy Project, Pennsylvania State University
• Michael Heuer heuermh@berkeley.edu, UC Berkeley AMPLab
• Andrey Kartashov Andrey.Kartashov@cchmc.org, Cincinnati Children's Hospital
• Dan Leehr dan.leehr@duke.edu, Duke University
• Hervé Ménager herve.menager@gmail.com, Institut Pasteur
• Maya Nedeljkovich maja.nedeljkovic@sbgenomics.com, Seven Bridges Genomics
• Matt Scales mscales@icr.ac.uk, Institute of Cancer Research, London
• Stian Soiland-Reyes soiland-reyes@cs.manchester.ac.uk, University of Manchester
• Luka Stojanovic luka.stojanovic@sbgenomics.com, Seven Bridges Genomics

Abstract §

A Command Line Tool is a non-interactive executable program that reads some input, performs a computation, and terminates after producing some output. Command line programs are a flexible unit of code sharing and reuse, unfortunately the syntax and input/output semantics among command line programs is extremely heterogeneous. A common layer for describing the syntax and semantics of programs can reduce this incidental complexity by providing a consistent way to connect programs together. This specification defines the Common Workflow Language (CWL) Command Line Tool Description, a vendor-neutral standard for describing the syntax and input/output semantics of command line programs.

Status of this document §

This document is the product of the Common Workflow Language working group. The latest version of this document is available in the "v1.0" directory at

https://github.com/common-workflow-language/common-workflow-language

The products of the CWL working group (including this document) are made available under the terms of the Apache License, version 2.0.

Table of contents

1. Introduction §

The Common Workflow Language (CWL) working group is an informal, multi-vendor working group consisting of various organizations and individuals that have an interest in portability of data analysis workflows. The goal is to create specifications like this one that enable data scientists to describe analysis tools and workflows that are powerful, easy to use, portable, and support reproducibility.

1.1 Introduction to CWL Command Line Tool standard v1.0.2 §

This specification represents the third stable release from the CWL group. Since the initial v1.0 release, v1.0.2 introduces the following updates to the CWL Command Line Tool standard. Documents should continue to use cwlVersion: v1.0 and existing v1.0 documents remain valid, however CWL documents that relied on previously undefined or underspecified behavior may have slightly different behavior in v1.0.2.

• 13 July 2016: Mark baseCommand as optional and update descriptive text.
• 14 November 2016: Clarify SoftwareRequirement spec fields.
• 12 March 2017:
  • Mark default as not required for link checking.
  • Add note that files in InitialWorkDir must have path in output directory.
  • Add note that writable: true applies recursively.
• 23 July 2017: (v1.0.1)
  • Add clarification about scattering over empty arrays.
  • Clarify interpretation of secondaryFiles on inputs.
  • Expanded discussion of semantics of File and Directory types
  • Fixed typo "EMACScript" to "ECMAScript"
  • Clarified application of input parameter default values when the input is null or undefined.
  • Clarified valid types and meaning of the format field on inputs versus outputs
  • Clarify that command line arguments must not interpreted as shell except when shellQuote: false
  • Clarify behavior of entryname
• 10 August 2017: (v1.0.2)
  • Clarify behavior resolving expressions in secondaryFile

Since draft-3, v1.0 introduces the following changes and additions to the CWL Command Line Tool standard:

• The Directory type.

• Syntax simplifcations: denoted by the map<> syntax. Example: inputs contains a list of items, each with an id. Now one can specify a mapping of that identifier to the corresponding CommandInputParameter.

  inputs:
   - id: one
     type: string
     doc: First input parameter
   - id: two
     type: int
     doc: Second input parameter
  

  can be

  inputs:
   one:
    type: string
    doc: First input parameter
   two:
    type: int
    doc: Second input parameter
  

• InitialWorkDirRequirement: list of files and subdirectories to be present in the output directory prior to execution.

• Shortcuts for specifying the standard output and/or error streams as a (streamable) File output.

• SoftwareRequirement for describing software dependencies of a tool.

• The common description field has been renamed to doc.

1.2 Purpose §

Standalone programs are a flexible and interoperable form of code reuse. Unlike monolithic applications, applications and analysis workflows which are composed of multiple separate programs can be written in multiple languages and execute concurrently on multiple hosts. However, POSIX does not dictate computer-readable grammar or semantics for program input and output, resulting in extremely heterogeneous command line grammar and input/output semantics among program. This is a particular problem in distributed computing (multi-node compute clusters) and virtualized environments (such as Docker containers) where it is often necessary to provision resources such as input files before executing the program.

Often this gap is filled by hard coding program invocation and implicitly assuming requirements will be met, or abstracting program invocation with wrapper scripts or descriptor documents. Unfortunately, where these approaches are application or platform specific it creates a significant barrier to reproducibility and portability, as methods developed for one platform must be manually ported to be used on new platforms. Similarly it creates redundant work, as wrappers for popular tools must be rewritten for each application or platform in use.

The Common Workflow Language Command Line Tool Description is designed to provide a common standard description of grammar and semantics for invoking programs used in data-intensive fields such as Bioinformatics, Chemistry, Physics, Astronomy, and Statistics. This specification defines a precise data and execution model for Command Line Tools that can be implemented on a variety of computing platforms, ranging from a single workstation to cluster, grid, cloud, and high performance computing platforms.

1.3 References to other specifications §

Javascript Object Notation (JSON): http://json.org

JSON Linked Data (JSON-LD): http://json-ld.org

YAML: http://yaml.org

Avro: https://avro.apache.org/docs/1.8.1/spec.html

Uniform Resource Identifier (URI) Generic Syntax: https://tools.ietf.org/html/rfc3986)

Internationalized Resource Identifiers (IRIs): https://tools.ietf.org/html/rfc3987

Portable Operating System Interface (POSIX.1-2008): http://pubs.opengroup.org/onlinepubs/9699919799/

Resource Description Framework (RDF): http://www.w3.org/RDF/

1.4 Scope §

This document describes CWL syntax, execution, and object model. It is not intended to document a CWL specific implementation, however it may serve as a reference for the behavior of conforming implementations.

1.5 Terminology §

The terminology used to describe CWL documents is defined in the Concepts section of the specification. The terms defined in the following list are used in building those definitions and in describing the actions of a CWL implementation:

may: Conforming CWL documents and CWL implementations are permitted but not required to behave as described.

must: Conforming CWL documents and CWL implementations are required to behave as described; otherwise they are in error.

error: A violation of the rules of this specification; results are undefined. Conforming implementations may detect and report an error and may recover from it.

fatal error: A violation of the rules of this specification; results are undefined. Conforming implementations must not continue to execute the current process and may report an error.

at user option: Conforming software may or must (depending on the modal verb in the sentence) behave as described; if it does, it must provide users a means to enable or disable the behavior described.

deprecated: Conforming software may implement a behavior for backwards compatibility. Portable CWL documents should not rely on deprecated behavior. Behavior marked as deprecated may be removed entirely from future revisions of the CWL specification.

2. Data model §

2.1 Data concepts §

An object is a data structure equivalent to the "object" type in JSON, consisting of a unordered set of name/value pairs (referred to here as fields) and where the name is a string and the value is a string, number, boolean, array, or object.

A document is a file containing a serialized object, or an array of objects.

A process is a basic unit of computation which accepts input data, performs some computation, and produces output data. Examples include CommandLineTools, Workflows, and ExpressionTools.

An input object is an object describing the inputs to an invocation of a process.

An output object is an object describing the output resulting from an invocation of a process.

An input schema describes the valid format (required fields, data types) for an input object.

An output schema describes the valid format for an output object.

Metadata is information about workflows, tools, or input items.

2.2 Syntax §

CWL documents must consist of an object or array of objects represented using JSON or YAML syntax. Upon loading, a CWL implementation must apply the preprocessing steps described in the Semantic Annotations for Linked Avro Data (SALAD) Specification. An implementation may formally validate the structure of a CWL document using SALAD schemas located at https://github.com/common-workflow-language/common-workflow-language/tree/master/v1.0

2.3 Identifiers §

If an object contains an id field, that is used to uniquely identify the object in that document. The value of the id field must be unique over the entire document. Identifiers may be resolved relative to either the document base and/or other identifiers following the rules are described in the Schema Salad specification.

An implementation may choose to only honor references to object types for which the id field is explicitly listed in this specification.

2.4 Document preprocessing §

An implementation must resolve $import and $include directives as described in the Schema Salad specification.

Another transformation defined in Schema salad is simplification of data type definitions. Type <T> ending with ? should be transformed to [<T>, "null"]. Type <T> ending with [] should be transformed to {"type": "array", "items": <T>}

2.5 Extensions and metadata §

Input metadata (for example, a lab sample identifier) may be represented within a tool or workflow using input parameters which are explicitly propagated to output. Future versions of this specification may define additional facilities for working with input/output metadata.

Implementation extensions not required for correct execution (for example, fields related to GUI presentation) and metadata about the tool or workflow itself (for example, authorship for use in citations) may be provided as additional fields on any object. Such extensions fields must use a namespace prefix listed in the $namespaces section of the document as described in the Schema Salad specification.

Implementation extensions which modify execution semantics must be listed in the requirements field.

3. Execution model §

3.1 Execution concepts §

A parameter is a named symbolic input or output of process, with an associated datatype or schema. During execution, values are assigned to parameters to make the input object or output object used for concrete process invocation.

A CommandLineTool is a process characterized by the execution of a standalone, non-interactive program which is invoked on some input, produces output, and then terminates.

A workflow is a process characterized by multiple subprocess steps, where step outputs are connected to the inputs of downstream steps to form a directed acylic graph, and independent steps may run concurrently.

A runtime environment is the actual hardware and software environment when executing a command line tool. It includes, but is not limited to, the hardware architecture, hardware resources, operating system, software runtime (if applicable, such as the specific Python interpreter or the specific Java virtual machine), libraries, modules, packages, utilities, and data files required to run the tool.

A workflow platform is a specific hardware and software implementation capable of interpreting CWL documents and executing the processes specified by the document. The responsibilities of the workflow platform may include scheduling process invocation, setting up the necessary runtime environment, making input data available, invoking the tool process, and collecting output.

A workflow platform may choose to only implement the Command Line Tool Description part of the CWL specification.

It is intended that the workflow platform has broad leeway outside of this specification to optimize use of computing resources and enforce policies not covered by this specification. Some areas that are currently out of scope for CWL specification but may be handled by a specific workflow platform include:

• Data security and permissions
• Scheduling tool invocations on remote cluster or cloud compute nodes.
• Using virtual machines or operating system containers to manage the runtime (except as described in DockerRequirement).
• Using remote or distributed file systems to manage input and output files.
• Transforming file paths.
• Determining if a process has previously been executed, and if so skipping it and reusing previous results.
• Pausing, resuming or checkpointing processes or workflows.

Conforming CWL processes must not assume anything about the runtime environment or workflow platform unless explicitly declared though the use of process requirements.

3.2 Generic execution process §

The generic execution sequence of a CWL process (including workflows and command line line tools) is as follows.

1. Load, process and validate a CWL document, yielding a process object.
2. Load input object.
3. Validate the input object against the inputs schema for the process.
4. Validate process requirements are met.
5. Perform any further setup required by the specific process type.
6. Execute the process.
7. Capture results of process execution into the output object.
8. Validate the output object against the outputs schema for the process.
9. Report the output object to the process caller.

3.3 Requirements and hints §

A process requirement modifies the semantics or runtime environment of a process. If an implementation cannot satisfy all requirements, or a requirement is listed which is not recognized by the implementation, it is a fatal error and the implementation must not attempt to run the process, unless overridden at user option.

A hint is similar to a requirement; however, it is not an error if an implementation cannot satisfy all hints. The implementation may report a warning if a hint cannot be satisfied.

Requirements are inherited. A requirement specified in a Workflow applies to all workflow steps; a requirement specified on a workflow step will apply to the process implementation of that step and any of its substeps.

If the same process requirement appears at different levels of the workflow, the most specific instance of the requirement is used, that is, an entry in requirements on a process implementation such as CommandLineTool will take precedence over an entry in requirements specified in a workflow step, and an entry in requirements on a workflow step takes precedence over the workflow. Entries in hints are resolved the same way.

Requirements override hints. If a process implementation provides a process requirement in hints which is also provided in requirements by an enclosing workflow or workflow step, the enclosing requirements takes precedence.

3.4 Parameter references §

Parameter references are denoted by the syntax $(...) and may be used in any field permitting the pseudo-type Expression, as specified by this document. Conforming implementations must support parameter references. Parameter references use the following subset of Javascript/ECMAScript 5.1 syntax, but they are designed to not require a Javascript engine for evaluation.

In the following BNF grammar, character classes, and grammar rules are denoted in '{}', '-' denotes exclusion from a character class, '(())' denotes grouping, '|' denotes alternates, trailing '*' denotes zero or more repeats, '+' denote one or more repeats, '/' escapes these special characters, and all other characters are literal values.

symbol::	{Unicode alphanumeric}+
singleq::	[' (( {character - '} | \' ))* ']
doubleq::	[" (( {character - "} | \" ))* "]
index::	[ {decimal digit}+ ]
segment::	. {symbol} | {singleq} | {doubleq} | {index}
parameter reference::	$( {symbol} {segment}*)

Use the following algorithm to resolve a parameter reference:

1. Match the leading symbol as the key
2. Look up the key in the parameter context (described below) to get the current value. It is an error if the key is not found in the parameter context.
3. If there are no subsequent segments, terminate and return current value
4. Else, match the next segment
5. Extract the symbol, string, or index from the segment as the key
6. Look up the key in current value and assign as new current value. If the key is a symbol or string, the current value must be an object. If the key is an index, the current value must be an array or string. It is an error if the key does not match the required type, or the key is not found or out of range.
7. Repeat steps 3-6

The root namespace is the parameter context. The following parameters must be provided:

• inputs: The input object to the current Process.
• self: A context-specific value. The contextual values for 'self' are documented for specific fields elsewhere in this specification. If a contextual value of 'self' is not documented for a field, it must be 'null'.
• runtime: An object containing configuration details. Specific to the process type. An implementation may provide opaque strings for any or all fields of runtime. These must be filled in by the platform after processing the Tool but before actual execution. Parameter references and expressions may only use the literal string value of the field and must not perform computation on the contents, except where noted otherwise.

If the value of a field has no leading or trailing non-whitespace characters around a parameter reference, the effective value of the field becomes the value of the referenced parameter, preserving the return type.

If the value of a field has non-whitespace leading or trailing characters around a parameter reference, it is subject to string interpolation. The effective value of the field is a string containing the leading characters, followed by the string value of the parameter reference, followed by the trailing characters. The string value of the parameter reference is its textual JSON representation with the following rules:

• Leading and trailing quotes are stripped from strings
• Objects entries are sorted by key

Multiple parameter references may appear in a single field. This case must be treated as a string interpolation. After interpolating the first parameter reference, interpolation must be recursively applied to the trailing characters to yield the final string value.

3.5 Expressions §

An expression is a fragment of Javascript/ECMAScript 5.1 code evaluated by the workflow platform to affect the inputs, outputs, or behavior of a process. In the generic execution sequence, expressions may be evaluated during step 5 (process setup), step 6 (execute process), and/or step 7 (capture output). Expressions are distinct from regular processes in that they are intended to modify the behavior of the workflow itself rather than perform the primary work of the workflow.

To declare the use of expressions, the document must include the process requirement InlineJavascriptRequirement. Expressions may be used in any field permitting the pseudo-type Expression, as specified by this document.

Expressions are denoted by the syntax $(...) or ${...}. A code fragment wrapped in the $(...) syntax must be evaluated as a ECMAScript expression. A code fragment wrapped in the ${...} syntax must be evaluated as a ECMAScript function body for an anonymous, zero-argument function. Expressions must return a valid JSON data type: one of null, string, number, boolean, array, object. Other return values must result in a permanentFailure. Implementations must permit any syntactically valid Javascript and account for nesting of parenthesis or braces and that strings that may contain parenthesis or braces when scanning for expressions.

The runtime must include any code defined in the "expressionLib" field of InlineJavascriptRequirement prior to executing the actual expression.

Before executing the expression, the runtime must initialize as global variables the fields of the parameter context described above.

The effective value of the field after expression evaluation follows the same rules as parameter references discussed above. Multiple expressions may appear in a single field.

Expressions must be evaluated in an isolated context (a "sandbox") which permits no side effects to leak outside the context. Expressions also must be evaluated in Javascript strict mode.

The order in which expressions are evaluated is undefined except where otherwise noted in this document.

An implementation may choose to implement parameter references by evaluating as a Javascript expression. The results of evaluating parameter references must be identical whether implemented by Javascript evaluation or some other means.

Implementations may apply other limits, such as process isolation, timeouts, and operating system containers/jails to minimize the security risks associated with running untrusted code embedded in a CWL document.

Exceptions thrown from an expression must result in a permanentFailure of the process.

3.6 Executing CWL documents as scripts §

By convention, a CWL document may begin with #!/usr/bin/env cwl-runner and be marked as executable (the POSIX "+x" permission bits) to enable it to be executed directly. A workflow platform may support this mode of operation; if so, it must provide cwl-runner as an alias for the platform's CWL implementation.

A CWL input object document may similarly begin with #!/usr/bin/env cwl-runner and be marked as executable. In this case, the input object must include the field cwl:tool supplying an IRI to the default CWL document that should be executed using the fields of the input object as input parameters.

3.7 Discovering CWL documents on a local filesystem §

To discover CWL documents look in the following locations:

/usr/share/commonwl/

/usr/local/share/commonwl/

$XDG_DATA_HOME/commonwl/ (usually $HOME/.local/share/commonwl)

$XDG_DATA_HOME is from the XDG Base Directory Specification

4. Running a Command §

To accommodate the enormous variety in syntax and semantics for input, runtime environment, invocation, and output of arbitrary programs, a CommandLineTool defines an "input binding" that describes how to translate abstract input parameters to an concrete program invocation, and an "output binding" that describes how to generate output parameters from program output.

4.1 Input binding §

The tool command line is built by applying command line bindings to the input object. Bindings are listed either as part of an input parameter using the inputBinding field, or separately using the arguments field of the CommandLineTool.

The algorithm to build the command line is as follows. In this algorithm, the sort key is a list consisting of one or more numeric or string elements. Strings are sorted lexicographically based on UTF-8 encoding.

1. Collect CommandLineBinding objects from arguments. Assign a sorting key [position, i] where position is CommandLineBinding.position and i is the index in the arguments list.

2. Collect CommandLineBinding objects from the inputs schema and associate them with values from the input object. Where the input type is a record, array, or map, recursively walk the schema and input object,

collecting nested `CommandLineBinding` objects and associating them with values from the input object.

3. Create a sorting key by taking the value of the position field at each level leading to each leaf binding object. If position is not specified, it is not added to the sorting key. For bindings on arrays and maps, the sorting key must include the array index or map key following the position. If and only if two bindings have the same sort key, the tie must be broken using the ordering of the field or parameter name immediately containing the leaf binding.

4. Sort elements using the assigned sorting keys. Numeric entries sort before strings.

5. In the sorted order, apply the rules defined in CommandLineBinding to convert bindings to actual command line elements.

6. Insert elements from baseCommand at the beginning of the command line.

4.2 Runtime environment §

All files listed in the input object must be made available in the runtime environment. The implementation may use a shared or distributed file system or transfer files via explicit download to the host. Implementations may choose not to provide access to files not explicitly specified in the input object or process requirements.

Output files produced by tool execution must be written to the designated output directory. The initial current working directory when executing the tool must be the designated output directory.

Files may also be written to the designated temporary directory. This directory must be isolated and not shared with other processes. Any files written to the designated temporary directory may be automatically deleted by the workflow platform immediately after the tool terminates.

For compatibility, files may be written to the system temporary directory which must be located at /tmp. Because the system temporary directory may be shared with other processes on the system, files placed in the system temporary directory are not guaranteed to be deleted automatically. A tool must not use the system temporary directory as a backchannel communication with other tools. It is valid for the system temporary directory to be the same as the designated temporary directory.

When executing the tool, the tool must execute in a new, empty environment with only the environment variables described below; the child process must not inherit environment variables from the parent process except as specified or at user option.

• HOME must be set to the designated output directory.
• TMPDIR must be set to the designated temporary directory.
• PATH may be inherited from the parent process, except when run in a container that provides its own PATH.
• Variables defined by EnvVarRequirement
• The default environment of the container, such as when using DockerRequirement

An implementation may forbid the tool from writing to any location in the runtime environment file system other than the designated temporary directory, system temporary directory, and designated output directory. An implementation may provide read-only input files, and disallow in-place update of input files. The designated temporary directory, system temporary directory and designated output directory may each reside on different mount points on different file systems.

An implementation may forbid the tool from directly accessing network resources. Correct tools must not assume any network access. Future versions of the specification may incorporate optional process requirements that describe the networking needs of a tool.

The runtime section available in parameter references and expressions contains the following fields. As noted earlier, an implementation may perform deferred resolution of runtime fields by providing opaque strings for any or all of the following fields; parameter references and expressions may only use the literal string value of the field and must not perform computation on the contents.

• runtime.outdir: an absolute path to the designated output directory
• runtime.tmpdir: an absolute path to the designated temporary directory
• runtime.cores: number of CPU cores reserved for the tool process
• runtime.ram: amount of RAM in mebibytes (2**20) reserved for the tool process
• runtime.outdirSize: reserved storage space available in the designated output directory
• runtime.tmpdirSize: reserved storage space available in the designated temporary directory

For cores, ram, outdirSize and tmpdirSize, if an implementation can't provide the actual number of reserved cores during the expression evaluation time, it should report back the minimal requested amount.

See ResourceRequirement for details on how to describe the hardware resources required by a tool.

The standard input stream and standard output stream may be redirected as described in the stdin and stdout fields.

4.3 Execution §

Once the command line is built and the runtime environment is created, the actual tool is executed.

The standard error stream and standard output stream (unless redirected by setting stdout or stderr) may be captured by platform logging facilities for storage and reporting.

Tools may be multithreaded or spawn child processes; however, when the parent process exits, the tool is considered finished regardless of whether any detached child processes are still running. Tools must not require any kind of console, GUI, or web based user interaction in order to start and run to completion.

The exit code of the process indicates if the process completed successfully. By convention, an exit code of zero is treated as success and non-zero exit codes are treated as failure. This may be customized by providing the fields successCodes, temporaryFailCodes, and permanentFailCodes. An implementation may choose to default unspecified non-zero exit codes to either temporaryFailure or permanentFailure.

4.4 Output binding §

If the output directory contains a file named "cwl.output.json", that file must be loaded and used as the output object. Otherwise, the output object must be generated by walking the parameters listed in outputs and applying output bindings to the tool output. Output bindings are associated with output parameters using the outputBinding field. See CommandOutputBinding for details.

5. CommandLineTool §

This defines the schema of the CWL Command Line Tool Description document.

Fields

5.1 CommandInputParameter §

An input parameter for a CommandLineTool.

Fields

5.1.1 Expression §

'Expression' is not a real type. It indicates that a field must allow runtime parameter references. If InlineJavascriptRequirement is declared and supported by the platform, the field must also allow Javascript expressions.

Symbols

symbol	description
ExpressionPlaceholder

5.1.2 CommandLineBinding §

When listed under inputBinding in the input schema, the term "value" refers to the the corresponding value in the input object. For binding objects listed in CommandLineTool.arguments, the term "value" refers to the effective value after evaluating valueFrom.

The binding behavior when building the command line depends on the data type of the value. If there is a mismatch between the type described by the input schema and the effective value, such as resulting from an expression evaluation, an implementation must use the data type of the effective value.

• string: Add prefix and the string to the command line.

• number: Add prefix and decimal representation to command line.

• boolean: If true, add prefix to the command line. If false, add nothing.

• File: Add prefix and the value of File.path to the command line.

• Directory: Add prefix and the value of Directory.path to the command line.

• array: If itemSeparator is specified, add prefix and the join the array into a single string with itemSeparator separating the items. Otherwise first add prefix, then recursively process individual elements. If the array is empty, it does not add anything to command line.

• object: Add prefix only, and recursively add object fields for which inputBinding is specified.

• null: Add nothing.

Fields

5.1.3 Any §

The Any type validates for any non-null value.

Symbols

symbol	description
Any

5.1.4 CWLType §

Extends primitive types with the concept of a file and directory as a builtin type.

Symbols

symbol	description
null	no value
boolean	a binary value
int	32-bit signed integer
long	64-bit signed integer
float	single precision (32-bit) IEEE 754 floating-point number
double	double precision (64-bit) IEEE 754 floating-point number
string	Unicode character sequence
null	no value
boolean	a binary value
int	32-bit signed integer
long	64-bit signed integer
float	single precision (32-bit) IEEE 754 floating-point number
double	double precision (64-bit) IEEE 754 floating-point number
string	Unicode character sequence
File	A File object
Directory	A Directory object

5.1.5 File §

Represents a file (or group of files when secondaryFiles is provided) that will be accessible by tools using standard POSIX file system call API such as open(2) and read(2).

Files are represented as objects with class of File. File objects have a number of properties that provide metadata about the file.

The location property of a File is a URI that uniquely identifies the file. Implementations must support the file:// URI scheme and may support other schemes such as http://. The value of location may also be a relative reference, in which case it must be resolved relative to the URI of the document it appears in. Alternately to location, implementations must also accept the path property on File, which must be a filesystem path available on the same host as the CWL runner (for inputs) or the runtime environment of a command line tool execution (for command line tool outputs).

If no location or path is specified, a file object must specify contents with the UTF-8 text content of the file. This is a "file literal". File literals do not correspond to external resources, but are created on disk with contents with when needed for a executing a tool. Where appropriate, expressions can return file literals to define new files on a runtime. The maximum size of contents is 64 kilobytes.

The basename property defines the filename on disk where the file is staged. This may differ from the resource name. If not provided, basename must be computed from the last path part of location and made available to expressions.

The secondaryFiles property is a list of File or Directory objects that must be staged in the same directory as the primary file. It is an error for file names to be duplicated in secondaryFiles.

The size property is the size in bytes of the File. It must be computed from the resource and made available to expressions. The checksum field contains a cryptographic hash of the file content for use it verifying file contents. Implementations may, at user option, enable or disable computation of the checksum field for performance or other reasons. However, the ability to compute output checksums is required to pass the CWL conformance test suite.

When executing a CommandLineTool, the files and secondary files may be staged to an arbitrary directory, but must use the value of basename for the filename. The path property must be file path in the context of the tool execution runtime (local to the compute node, or within the executing container). All computed properties should be available to expressions. File literals also must be staged and path must be set.

When collecting CommandLineTool outputs, glob matching returns file paths (with the path property) and the derived properties. This can all be modified by outputEval. Alternately, if the file cwl.output.json is present in the output, outputBinding is ignored.

File objects in the output must provide either a location URI or a path property in the context of the tool execution runtime (local to the compute node, or within the executing container).

When evaluating an ExpressionTool, file objects must be referenced via location (the expression tool does not have access to files on disk so path is meaningless) or as file literals. It is legal to return a file object with an existing location but a different basename. The loadContents field of ExpressionTool inputs behaves the same as on CommandLineTool inputs, however it is not meaningful on the outputs.

An ExpressionTool may forward file references from input to output by using the same value for location.

Fields

5.1.5.1 Directory §

Represents a directory to present to a command line tool.

Directories are represented as objects with class of Directory. Directory objects have a number of properties that provide metadata about the directory.

The location property of a Directory is a URI that uniquely identifies the directory. Implementations must support the file:// URI scheme and may support other schemes such as http://. Alternately to location, implementations must also accept the path property on Directory, which must be a filesystem path available on the same host as the CWL runner (for inputs) or the runtime environment of a command line tool execution (for command line tool outputs).

A Directory object may have a listing field. This is a list of File and Directory objects that are contained in the Directory. For each entry in listing, the basename property defines the name of the File or Subdirectory when staged to disk. If listing is not provided, the implementation must have some way of fetching the Directory listing at runtime based on the location field.

If a Directory does not have location, it is a Directory literal. A Directory literal must provide listing. Directory literals must be created on disk at runtime as needed.

The resources in a Directory literal do not need to have any implied relationship in their location. For example, a Directory listing may contain two files located on different hosts. It is the responsibility of the runtime to ensure that those files are staged to disk appropriately. Secondary files associated with files in listing must also be staged to the same Directory.

When executing a CommandLineTool, Directories must be recursively staged first and have local values of path assigend.

Directory objects in CommandLineTool output must provide either a location URI or a path property in the context of the tool execution runtime (local to the compute node, or within the executing container).

An ExpressionTool may forward file references from input to output by using the same value for location.

Name conflicts (the same basename appearing multiple times in listing or in any entry in secondaryFiles in the listing) is a fatal error.

Fields

5.1.6 CommandInputRecordSchema §

Fields

5.1.7 CommandInputRecordField §

Fields

5.1.7.1 CommandInputEnumSchema §

Fields

5.1.7.2 CommandInputArraySchema §

Fields

5.2 CommandOutputParameter §

An output parameter for a CommandLineTool.

Fields

5.2.1 stdout §

Only valid as a type for a CommandLineTool output with no outputBinding set.

The following

outputs:
  an_output_name:
    type: stdout

stdout: a_stdout_file

is equivalent to

outputs:
  an_output_name:
    type: File
    streamable: true
    outputBinding:
      glob: a_stdout_file

stdout: a_stdout_file

If there is no stdout name provided, a random filename will be created. For example, the following

outputs:
  an_output_name:
    type: stdout

is equivalent to

outputs:
  an_output_name:
    type: File
    streamable: true
    outputBinding:
      glob: random_stdout_filenameABCDEFG

stdout: random_stdout_filenameABCDEFG

Symbols

symbol	description
stdout

5.2.2 stderr §

Only valid as a type for a CommandLineTool output with no outputBinding set.

The following

outputs:
  an_output_name:
  type: stderr

stderr: a_stderr_file

is equivalent to

outputs:
  an_output_name:
    type: File
    streamable: true
    outputBinding:
      glob: a_stderr_file

stderr: a_stderr_file

If there is no stderr name provided, a random filename will be created. For example, the following

outputs:
  an_output_name:
    type: stderr

is equivalent to

outputs:
  an_output_name:
    type: File
    streamable: true
    outputBinding:
      glob: random_stderr_filenameABCDEFG

stderr: random_stderr_filenameABCDEFG

Symbols

symbol	description
stderr

5.2.3 CommandOutputBinding §

Describes how to generate an output parameter based on the files produced by a CommandLineTool.

The output parameter value is generated by applying these operations in the following order:

• glob
• loadContents
• outputEval
• secondaryFiles

Fields

5.2.4 CommandOutputRecordSchema §

Fields

5.2.5 CommandOutputRecordField §

Fields

5.2.5.1 CommandOutputEnumSchema §

Fields

5.2.5.2 CommandOutputArraySchema §

Fields

5.3 InlineJavascriptRequirement §

Indicates that the workflow platform must support inline Javascript expressions. If this requirement is not present, the workflow platform must not perform expression interpolatation.

Fields

5.4 SchemaDefRequirement §

This field consists of an array of type definitions which must be used when interpreting the inputs and outputs fields. When a type field contain a IRI, the implementation must check if the type is defined in schemaDefs and use that definition. If the type is not found in schemaDefs, it is an error. The entries in schemaDefs must be processed in the order listed such that later schema definitions may refer to earlier schema definitions.

Fields

5.4.1 InputRecordSchema §

Fields

5.4.2 InputRecordField §

Fields

5.4.2.1 InputEnumSchema §

Fields

5.4.2.2 InputArraySchema §

Fields

5.5 DockerRequirement §

Indicates that a workflow component should be run in a Docker container, and specifies how to fetch or build the image.

If a CommandLineTool lists DockerRequirement under hints (or requirements), it may (or must) be run in the specified Docker container.

The platform must first acquire or install the correct Docker image as specified by dockerPull, dockerImport, dockerLoad or dockerFile.

The platform must execute the tool in the container using docker run with the appropriate Docker image and tool command line.

The workflow platform may provide input files and the designated output directory through the use of volume bind mounts. The platform should rewrite file paths in the input object to correspond to the Docker bind mounted locations. That is, the platform should rewrite values in the parameter context such as runtime.outdir, runtime.tmpdir and others to be valid paths within the container.

When running a tool contained in Docker, the workflow platform must not assume anything about the contents of the Docker container, such as the presence or absence of specific software, except to assume that the generated command line represents a valid command within the runtime environment of the container.

Interaction with other requirements §

If EnvVarRequirement is specified alongside a DockerRequirement, the environment variables must be provided to Docker using --env or --env-file and interact with the container's preexisting environment as defined by Docker.

Fields

5.6 SoftwareRequirement §

A list of software packages that should be configured in the environment of the defined process.

Fields

5.7 SoftwarePackage §

Fields

5.8 InitialWorkDirRequirement §

Define a list of files and subdirectories that must be created by the workflow platform in the designated output directory prior to executing the command line tool.

Fields

5.8.1 Dirent §

Define a file or subdirectory that must be placed in the designated output directory prior to executing the command line tool. May be the result of executing an expression, such as building a configuration file from a template.

Fields

5.9 EnvVarRequirement §

Define a list of environment variables which will be set in the execution environment of the tool. See EnvironmentDef for details.

Fields

5.10 EnvironmentDef §

Define an environment variable that will be set in the runtime environment by the workflow platform when executing the command line tool. May be the result of executing an expression, such as getting a parameter from input.

Fields

5.11 ShellCommandRequirement §

Modify the behavior of CommandLineTool to generate a single string containing a shell command line. Each item in the argument list must be joined into a string separated by single spaces and quoted to prevent intepretation by the shell, unless CommandLineBinding for that argument contains shellQuote: false. If shellQuote: false is specified, the argument is joined into the command string without quoting, which allows the use of shell metacharacters such as | for pipes.

Fields

5.12 ResourceRequirement §

Specify basic hardware resource requirements.

"min" is the minimum amount of a resource that must be reserved to schedule a job. If "min" cannot be satisfied, the job should not be run.

"max" is the maximum amount of a resource that the job shall be permitted to use. If a node has sufficient resources, multiple jobs may be scheduled on a single node provided each job's "max" resource requirements are met. If a job attempts to exceed its "max" resource allocation, an implementation may deny additional resources, which may result in job failure.

If "min" is specified but "max" is not, then "max" == "min" If "max" is specified by "min" is not, then "min" == "max".

It is an error if max < min.

It is an error if the value of any of these fields is negative.

If neither "min" nor "max" is specified for a resource, an implementation may provide a default.

Fields

5.13 CWLVersion §

Version symbols for published CWL document versions.

Symbols

symbol	description
draft-2
draft-3.dev1
draft-3.dev2
draft-3.dev3
draft-3.dev4
draft-3.dev5
draft-3
draft-4.dev1
draft-4.dev2
draft-4.dev3
v1.0.dev4
v1.0