modules

 1	<?xml version="1.0" encoding="UTF-8" ?>
 2	<x:schema xmlns:x="http://www.w3.org/2001/XMLSchema" xmlns:vc="http://www.w3.org/2007/XMLSchema-versioning" vc:minVersion="1.1" xmlns:xerces="http://xerces.apache.org">
 3	
 4		<!-- definition of the task parameters -->
 5		<x:complexType name="sleepTaskParameterType">
 6			<x:all>
 7				<x:element name="wait" type="paramWait_sleep" minOccurs="1" maxOccurs="1" />
 8			</x:all>
 9		</x:complexType>
10		
11		...
12	
13		<!-- make task definition availible via substitution group -->
14		<x:element name="sleepTask" type="sleepTaskType" substitutionGroup="abstractTask" />
15	
16		<!-- module specific parameter types -->
17		<x:complexType name="paramWait_sleep">
18			<x:simpleContent>
19				<x:restriction base="paramString">
20					<x:assertion test="matches($value, '(${[A-Za-z_]+})|($(.+))|([[({]($[A-Za-z_]+(,s*){0,1}){0,1}([0-9]+(,S*){0,1}){0,1}[])}])') or matches($value, '^[0-9]+[smhd]{0,1}$')" xerces:message="Parameter with name '{$tag}' must match [0-9]+[smhd]{0,1}." />
21				</x:restriction>
22			</x:simpleContent>
23		</x:complexType>
24	
25	</x:schema>

sleepTaskParameterType

wait

matches()

name

sleepTask

?Task

helper_scripts/configureExamples.sh

 1	<?xml version="1.0" encoding="UTF-8"?>
 2	<watchdog xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="watchdog.xsd" watchdogBase="{%INSTALL%}" isTemplate="true">
 3	
 4		<!-- begin task block and use that mail to inform the user on success or failure -->
 5		<tasks mail="{%MAIL%}">
 6	
 7			<!-- definition a simple sleep task -->
 8			<sleepTask id="1" name="sleep">
 9				<parameter>
10					<wait>30s</wait>
11				</parameter>
12			</sleepTask>
13		</tasks>
14	</watchdog>

sleep

watchdog

watchdogBase

isTemplate

tasks

id

name

parameter

flagName

/flagName

flagName

/flagName

paramName

value

/paramName

watchdog

watchdogBase

watchdog

isTemplate

false

tasks

[mail]

tasks

[projectName]

?Task

[id]

?Task

name

?Task

[processBlock]

?Task

[executor]

?Task

[environment]

?Task

[maxRunning]

?Task

[notify]

disabled

?Task

[checkpoint]

disabled

?Task

[confirmParam]

disabled

?Task

[version]

1

?Task

[posX]

?Task

[posY]

4 Detailed XML format explanation In the following sections the complete range of functions of Watchdog's XML format is explained. The following elements must be defined as child elements of the

settings

element before the

tasks

element begins and are valid within the complete XML file:

• processBlock

  - process a task with varying parameters (see 4.1)

• executors

  - define different executor environments (see 4.3)

• wrappers

  - execution wrappers usable as package managers or for virtualization (see 4.4)

• constants

  - defines constants that substitute placeholders (see 4.5)

• environments

  - define or update environment variables (see 4.6)

• modules

  - define multiple module include directories (see 4.11)

Apart from the

parameter

element, the following elements are allowed in

?Task

elements:

• environment

  - define or update environment variables (see 4.6)

• dependencies

  - define dependencies between tasks (see 4.2)

• streams

  - define location of standard streams and set a working directory (see 4.8)

• checkers

  - usage of custom success or error checkers (see 4.12)

• actions

  - define task actions that are performed before or after tasks execution (see 4.9)

4.1 Process blocks Watchdog is able to process multiple tasks of the same type, which differ only in some parameter values, without the need to define all of these tasks separately. This function is referred to as process blocks while the tasks created by an process block are called subtasks of the task. There are four different possibilities to define process blocks as childs of the

processBlock

element:

• processSequence

  - argument is numeric

• processFolder

  - argument is a path to a file

• processInput

  - multiple arguments obtained from dependencies

• processTable

  - multiple arguments stored in a tab-separated file with names of variables stored in the first line

When the

processBlock

attribute of a task is set the argument of the process folder or sequence is substituted at run time within

parameter

,

streams

,

checkers

,

actions

and

environment

elements in the following manner:

• processSequence

  - []/{}/() -> number

• processFolder

  - {} -> absolute path to the file

• processFolder

  - () -> absolute path to the parent folder of the file

• processFolder

  - [] -> name of the file

• processFolder

  - [

  n

  ]/{

  n

  } ->

  n

  suffixes of the filename are truncated using . as separator

• processFolder

  - (

  n

  ) ->

  n

  suffixes of the parent folder are truncated using / as separator

• processFolder

  - ([{

  n

  ,

  sep

  }]) -> suffixes of the value are truncated using

  sep

  as separator (might also be a regex)

• processTable

  - ([{

  $COL_NAME

  }]) -> value stored in the column named

  $COL_NAME

• processTable

  - ([{

  $COL_NAME

  ,

  n

  ,

  sep

  }]) -> value stored in the column named

  $COL_NAME

  but with suffix truncation as described above

• processInput

  - ([{

  $RET_NAME

  }]) -> return value of a dependency with the name

  $RET_NAME

• processInput

  - ([{

  $RET_NAME

  ,

  n

  ,

  sep

  }]) -> return value of a dependency with the name

  $RET_NAME

  but with suffix truncation as described above

If a task depends on two tasks, which return variables with the same name, the return value of the task with the smaller id will be overwritten. Deviating from this, return values from separate dependencies will overwrite the ones from global dependencies if both use the same name for a variable. Example 4 shows three different ways how process blocks can be specified. First a

processSequence

named

qualities

is defined that creates the numbers 1,3,5,7 and 9 (7). In the next line a

processFolder

is defined that will process all files stored in

{%EXAMPLE_DATA%}/spec

that end with

.spec

(8). The syntax which must be used in the

pattern

attribute is the same as in bash. If a

processFolder

is a child element of a

baseFolder

, the

folder

attribute of the

processFolder

will be prefixed with the

folder

attribute of the

baseFolder

(9-14). The attribute

disableExistenceCheck

that is enabled for the

processFolder

with the name

gzFiles

causes Watchdog not to force the existence of the folder when it is started (12).
The task with

id

1

will compress all

.txt

files in the folders

{%EXAMPLE_DATA%}/txt

and

{%EXAMPLE_DATA%}/other_txt

and store them in

{%EXAMPLE_DATA%}/txt_zipped

(20-25). Whereas, the task with

id

2

will compress a file with different quality values (28-40). The compressed files will be stored with

txtFile1_q

as prefix and the used quality as suffix in

{%EXAMPLE_DATA%}/qualityTest

(34). Additionally, an environment variable with the name

QUALITY

is set which also contains the set quality (38). The sleep task at the end of the example shows how the colums of a

processTable

can be used as input (43-57). If the variable is of numeric type it can also be used within simple calculations which are presented in 4.10 (55). 4.2 Dependencies By default all tasks specified in the XML document are independent from each other. That implies that all tasks are scheduled at the same time if no other constraints exist. It is possible to define dependencies between tasks using the

depends

element that expects as value the id or name of an already defined task. The element must be a child of a

dependencies

element. Without any arguments the task will not be scheduled until all (sub)tasks of the dependencies have finished successfully. By setting the

separate

argument to

true

a subtask can depend only on the corresponding subtask the task depends on. This option is only meaningful if both tasks are process block tasks and work on the same input set or a transformed version of it. In example 5 a

compress

task with id

2

is defined which depends on the before defined

sleep

task (23-32). Additionally, a task, which will decompress the compressed files immediately after the compression is finished, is defined (35-45). In order to achieve this behavior the

separate

attribute is set to

true

(43). Because the

.txt

ending of the original filename was cropped and a

.gz

ending was added, only the first part of the filename is considered as specified in the

prefixName

attribute (26,43). 4.3 Execution environments By default the tasks are executed one after the other locally on the host which runs Watchdog. It is possible to define different execution environments using the

executors

element. Possible environments:

• local

  - task is executed on the local host

• remote

  - task is executed on a remote host using ssh

• cluster

  - task is executed on a computer cluster using DRMAA

• sge

  - task is executed on a SGE computer cluster using control binaries

• slurm

  - task is executed on a SLURM computer cluster using control binaries

In example 6 four different execution environments are defined as childs of the

executors

element (6-11). Tasks scheduled on first executor will run on the host on which Watchdog was started (7).

The executor with the name

defaultCluster

is used by default and runs on the

short.q

queue of the computer cluster (8). Before the actual module command is executed on that executor, the commands stored in the before script

ulimitMemory.sh

are executed. If a relative path is given, the script must be located in

core_lib/executor_scripts

. In this example the script will enforce that the requested memory is not exceeded using the

ulimit

command.

The next executor is reserved for high performance tasks because it reserves 4 slots on the cluster with each slot consuming three gigabyte of main memory (9). In order not to occupy the complete computing power the attribute

maxRunning

is set to four which means that a maximum of four tasks will run simultaneously on that execution environment.

In line 10 an example for a remote executor is given which executes tasks via ssh using a host named

superComputer

.

Afterwards the same sleep task is defined as in the first example and will run on the local executor (16). The other executors can be tested once you adapted them to your local infrastructure (see 2.7 and 2.8). 4.4 Execution wrappers Execution wrappers can be used in combination with

?Executor

to support the use of package managers (e.g. Conda) or virtualization (e.g. Docker). Multiple execution wrappers can be defined within the

wrappers

element. Additional execution wrappers can be implemented using Watchdog's plugin system (see 7.2).

Implemented package managers:

• conda

  - tasks are executed in a Conda environment if supported by the module (see 5.7)

Implemented virtualizer:

• docker

  - tasks are executed in a Docker container

In example 7 a Conda execution wrapper is defined as child of the

wrappers

element (5-18). The wrapper will be used for tasks that were created from modules that support the Conda package manager wrapper (7). If a module does not support the Conda package manager, the package manager will be ignored during task execution. The attribute

path2conda

defines the conda binary to use. Conda environments are initialized once per module and stored in the

path2environments

directory.

In line 9 another Conda execution wrapper is defined, which is configured to be used within a Docker image. The Docker wrapper is defined with the

docker

element (11-17). The attribute

path2docker

must specify the path to the virtualizer binary, which can be docker, podman or singularity. The

image

attribute defines which image is used to start the container (e.g.

conda/miniconda3

; in case of singularity use

docker://conda/miniconda3

). If the image is not present in the local image store, it will be downloaded from the default registry by the virtualizer binary. A different image registry can be used with this syntax:

<registry-ip>:<registry-port>/image

By default module specific images are used instead of the image provided in the image

attribute

. Module specific image are identified from files matching the pattern '

docker.image(.v[0-9]+)?.name

' in the module directory, which contain only a single line with the image name. For different versions, different image files can be included (ending in '

.v[0-9]+.name

'). If no image name file for the module version used in a task is included in the module, the default image name file for the module ('

docker.image.name

') is used. If no image name file is included in the module, the image name provided in the image

attribute

is used. If a Docker wrapper should not use the module specific image names, the attribute

loadModuleSpecificImage

can be set to

false

.

Most tasks process files stored on the host file system. Hence, these files must be made available within the container using mount points. Watchdog will try to detect the required mount paths automatically using parameters und streams of a task. Hence, most workflows can be run in a Docker container without further adjustments. The element

blacklist

can be used to avoid mounting of automatically detected directories (e.g. directories mounted on local disks as

/tmp/

or

/usr/local/storage/

). Another possibility is to disable the automatic detection using the

disableAutodetectMount

attribute and to define the required mounts manually using

mount

(14-16).

In line 21, a localhost executor is defined that uses the previously defined execution wrapper named

condaWrapper

. Another local executor is defined in line 23, which uses the Docker wrapper

podman

and the Conda execution wrapper

condaContainer

. On both executors a simple sleep task will be executed (28-47). 4.5 Global constants A constant can be defined globally using the

const

elements which must be a child of a

constants

element. The parent element itself must be a child of the

settings

environment. Every

const

element must own a unique name which is set with the

name

attribute. The value of the constant is stored between the opening and closing element tag.

${NAME_OF_CONSTANT}

is substituted with the corresponding constant in every attribute or text content. Only the

watchdogBase

attribute of

watchdog

, the

default

attribute of

?Executor

and the

id

attribute of

?Task

and within

depends

elements can not be substituted.

Currently, there are two pre-defined constants. The first is named

${TMP}

and is substituted within

?Task

tags with the working directory of the executor that will execute the task. The second is named

${WF_PARENT}

and is replaced with the parent folder containing the XML workflow file. In example 8 three constants are defined (6-10). The constant named

${WAIT_TIME}

is used as wait time in the sleep task (21). The other two constants are used to construct the standard output file path (18). 4.6 Environment variables Some tools expect specific environment variables to be set correctly. For example the PATH variable is important because executable programs are located only in directories defined by that variable. The environment variables which are set on the host running Watchdog can be simply inherited. With help of the

var

element new variables can be defined or updated. The name of the variable must be defined with the

name

attribute while the value is stored between the opening and closing element tag. The parent element of each

var

element must be a

environment

element which also owns a

name

attribute. This

name

attribute is used to link the environment with a task using the

environment

attribute all tasks possess. It is also possible to define environment variables locally within task definitions. If local and global variables with the same name are set, the local ones override the global variables.

The following environment variables are set by Watchdog by default:

• IS_WATCHDOG_JOB: if module was executed by Watchdog this value is set to

  1

• WATCHDOG_CORES: number of reserved cores if task runs on a cluster environment
• WATCHDOG_MEMORY: number of total reserved memory in megabyte if task runs on a cluster environment

In example 9 an environment named

pathEnv

is defined in which the variable

PATH

is updated (8). The entry

~/software/bin

is added at the beginning of the

PATH

variable and after the default seperator character the previous value is kept. The

environment

attribute of the

env

task is set to the name of the previously defined environment (17). Additionally two local environment variables are defined (23-26). The first one replaces the default shell with

/bin/sh

while the second one updates a variable called

TEST

using an alternative separator. 4.7 Mail notification By default Watchdog informs the user only when an error occurs during the execution of a task or if an error was detected afterwards. But these behaviour can be changed using the

notify

attribute of tasks. In example 10 different notification options are presented. The first defined task is the simple sleep task from the previous examples for which the

notify

attribute is set to

enabled

(14). Once the task is finished a mail will be sent to the address the user specified in the

mail

attribute of the

tasks

element (12). The second defined task is based on a process block named

sleepTime

and causes Watchdog to inform the user as soon as a subtask is finished because the

notify

attribute is set to

subtask

(7, 21). 4.8 Standard streams and working directory By default the stdout and stderr stream of the tool which is executed by watchdog is not saved. This can be changed by setting a path via the

stdout

or

stderr

element. It is also possible to use a file as input via the

stdin

element. Additionally, a working directory can be set by using the

workingDir

element. When this is done also relative path for

stdout

,

stderr

and

stdin

are allowed. Other than usual, the elements must occour in the same order as they are listed in the following:

workingDir

,

stdout

,

stderr

and

stdin

(but each of them is optional) In example 11 the working directory is set to

/tmp

(14). Afterwards, the standard output of the

sleep

task is written to the file

{%EXAMPLE_DATA%}/sleepTest.out

(15). The standard error stream is appended at the end of a file named

/tmp/sleepTest.err

(16). 4.9 Task actions Additional operations can be executed before and after the actual task is executed. Currently a set of IO operations are implemented that can be used to roll out data the task depends on or to clean up once the task is finished. This feature is especically usefull when Watchdog is running in slave mode (see 4.3) in order to reduce load for the shared file system as some files might be requirements for different tasks. Hence, these files would have to be transfered multiple times from the shared file system to the host executing the task.
Moreover, files stored on remote files systems can be up- or downloaded by Watchdog. By default, virtual file systems based on the protocols File, HTTP, HTTPS, FTP, FTPS and SFTP as well as the main memory (RAM) are supported. These virtual file systems are provided by the Commons Virtual File System project of the Apache Software Foundation. Examples for valid URIs of these file systems can be can be found here. However, any file system with an implementation of the

FileProvider

can also be included by the user as described in 7.1.

Task actions are defined in an

actions

tag as child of

?Task

. Slave mode is automatically activated if a task action is used. Currently six different IO operations are implemented:

• createFile

  - creates an empty file

• createFolder

  - create an empty folder

• copyFile

  - copies a file

• copyFolder

  - copies a folder (with content)

• deleteFile

  - deletes a file

• deleteFolder

  - deletes a folder (with content)

The event after which the task actions are executed must be defined using the

time

attribute each

actions

tag owns. The following arguments are available:

• beforeTask

  - before the task is executed

• afterTask

  - after the task is executed

• onSuccess

  - when the task was successfully executed

• onFailure

  - when task execution failed

• beforeTerminate

  - before Watchdog or a slave terminates itself

In example 12 a file name

/tmp/watchdog_file_to_compress.tmp

is compressed using gzip (5-14). Before the compress task is executed, the task action defined within the

actions

tag is executed because the

time

attribute is set to

beforeTask

(11-13). The task action copies the file stored in

{%INSTALL%}/examples/example_task_actions.xml

to

/tmp/watchdog_file_to_compress.tmp

(12). 4.10 Simple calculations Within a

?Task

element simple calculations can be preformed using the

$(expr)

construct whereby

expr

must be a numerical equation. The following operators are supported: +, -, *, /, ^, ² and ³. Additional the brackets

()

are provided. Moreover in case of a

processSequence

i

is replaced by the current value of the process sequence. In the more general case of a

processBlock

x

is substituted by an increasing number starting at

1

. The result of all calculations is rounded to five decimal places or converted to an integer if it is one. In example 13 the usage of the

$(expr)

construct is shown. The wait time for the sleep task is calculated based on the input numbers of the

processSequence

(7, 16). In the second gzip task the number of the subtask is used to name the standard output files (23). 4.11 Multiple module search folders By default Watchdog tries to locate modules in a folder named

modules/

stored in the installation directory of Watchdog. By using the

modules

element as child of

settings

, additional folders can be added. In example 14 the default directory is changed to

myCustomFolder/

with the

defaultFolder

attribute (7). Moreover, in line 8 an additional folder is added to Watchdog's search path. Watchdog will now try to locate modules in the directories

{%INSTALL%}/myCustomFolder/

and

/home/additionalModules/

. In order to test that example you must create a new folder, copy the

sleep

module from Watchdog's module folder and adapt the path in line 7 or 8 to match that folder. 4.12 Custom success and error checker In some cases the exit code of a command is not a reliable indicator whether the command was executed successfully or not. For example some tools return as exit code zero regardless of whether the command succeeded or failed. Furthermore, a command could succeed technically but the desired result is not obtained (e.g. wrong index used for mapping of RNA-seq data results in a very low mapping rate). In order to handle such cases the user has the option to implement custom success and error checkers in Java that are executed by Watchdog once a task has terminated. Two steps must be performed to use custom checkers: implementation in Java and invocation in the XML workflow.

Interfaces for checkers are stored in the package de.lmu.ifi.bio.watchdog.interfaces. Basically a function returning a boolean value that indicates whether the task succeeded or failed must be implemented. The constructor must accept as first argument a object of the type Task that contains information about the task that was finished. Additional arguments of type Boolean, Integer, Double or String can be passed via the XML definition.

Example 15 shows an example of how an success checker can be added to a task by using the

checkers

element as a child of

?Task

(12-18). In addition to the location of the compiled Java class and the full class name arguments can be passed to the constructor of the class. In this example one variable of type

string

is passed to the constructor of the success checker using the

cArg

element (16). Once the task is finished, the checkers are evaluated in the same order as they were added in the XML workflow. In cases in which simultaneously success and error were detected, the task will be treated as failed. In this example the success checker will ensure that the file

{%INSTALL%}/examples/mail_config exists

and is not empty (15-16)