Creating Files at Runtime

Overview

Teaching: 10 min
Exercises: 0 min

Questions

• How do I create required input files from input parameters at runtime?

• How do I invoke a script rather than just a simple command line?

• Besides inputBinding, how else can I pass arguments to the tool?

Objectives

• Learn how to create files on the fly during runtime.

• Learn how to use expressions in bash scripts.

Sometimes you need to create a file on the fly from input parameters, such as tools which expect to read their input configuration from a file rather than the command line parameters, or need a small wrapper shell script.

To generate such files we can use the InitialWorkDirRequirement.

createfile.cwl

class: CommandLineTool
cwlVersion: v1.0
baseCommand: ["sh", "example.sh"]

requirements:
  InitialWorkDirRequirement:
    listing:
      - entryname: example.sh
        entry: |-
          PREFIX='Message is:'
          MSG="\${PREFIX} $(inputs.message)"
          echo \${MSG}

inputs:
  message: string
outputs:
  example_out:
    type: stdout
stdout: output.txt

Any expressions like $(inputs.message) are expanded by the CWL engine before creating the file; here inserting the value at the input message.

Tip

The CWL expressions are independent of any shell variables used later during command line tool invocation. That means that any genuine need for the character $ must be escaped with \, for instance \${PREFIX} above is expanded to ${PREFIX} in the generated file to be evaluated by the shell script instead of the CWL engine.

To test the above CWL tool use this job to provide the input value message:

echo-job.yml

message: Hello world!

Before we run this, lets look at each step in a little more detail. The base command baseCommand: ["sh", "example.sh"] will execute the command sh example.sh. This will run the file we create in the shell.

InitialWorkDirRequirement requires a listing. As the listing is a YAML array we need a - on the first line of each element of the array, in this case we have just one element. entryname: can have any value, but it must match what was specified in the baseCommand. The final part is entry:, this is followed by |- which is YAML quoting syntax, and means that you are using a multiline string (without it we would need to write the whole script on one line). (see the YAML Guide for more about the formating)

Now invoke cwl-runner with the tool wrapper and the input object on the command line:

$ cwl-runner createfile.cwl echo-job.yml
[job createfile.cwl] /private/tmp/docker_tmphrqxxcdl$ sh \
    example.sh > /private/tmp/docker_tmphrqxxcdl/output.txt
Could not collect memory usage, job ended before monitoring began.
[job createfile.cwl] completed success
{
    "example_out": {
        "location": "file:///home/example/output.txt",
        "basename": "output.txt",
        "class": "File",
        "checksum": "sha1$9045abe4bd04dd8ccfe50c6ff61820b784b64aa7",
        "size": 25,
        "path": "/home/example/output.txt"
    }
}
Final process status is success
$ cat output.txt
Message is: Hello world!

Key Points

• Use InitialWorkDirRequirement to specify input files that need to be created during tool runtime.