Task Phases

Each task in a codon workflow goes through five phases sequentially. Each task element belongs to one or more of these phases.

1. Action Phase

For a simple task the action attached to the task is called. Action is specified using the yml elements:

action: clients.jplaceholder.get_posts
input:
  userId: <%jmes main.userId %>

Here an argument map is created with the keys specified in input and values specified in the respective expressions. The argument map is passed to the action get_posts of HTTP client jplaceholder (http clients are specified in the endpoints section of config.yml). After the action is completed the response of the action is stored in a response map. For HTTP actions generated using swagger status code is also stored. If the status code was non-2XX (a failure) then the action is deemed as a failed. In case of custom action, if the action returns an non-nil error then the action is deemed as a failure.

2. Publish Phase

Once the response and status of the action is available, the task has to decide which variables to publish using the data collected. If the action was successful the variables declared in publish element are published to main variable bucket:

publish:
  - latest_post: <%jmes action.max_by([], &id) %>
  - latest_post_id: <%jmes main.latest_post.id %>

Variables are published sequentially. So a variable published to main bucket (like latest_post) is available for use in publishing subsequent variables (like latest_post_id).

Similarly we have publish-on-error which provides a list of variables published in case of a failure response from the action (or the HTTP client).

3. Next Phase

Once the desired variables are published, the task processor has to decide which subsequent tasks to call. In case the action was a success, the tasks specified in on-success are considered:

on-success:
  - get_comments: true
  - get_likes: <%jmes main.latest_post.has_likes %>

In the above example the tasks get_comments and get_likes are called concurrently. The task get_likes is called only when the expression <%jmes main.latest_post.has_likes %> evaluates to true and get_comments is always called. These expressions can only be boolean conditional expressions which decide whether the subsequent task is to be called.

The task executor then waits until all the tasks called concurrently are finished.

Similar to on-success we have on-error.

4. Complete Phase

After all the subsequent tasks from previous phase are finished, the task executor calls the tasks specified in on-complete element. These tasks are called in sequence (not concurrently) and regardless of whether action was success or failure. The structure of this element is same as on-success. This construct can be used to perform clean up tasks or perform tasks serially.

on-complete:
  - task1: <%jmes expr1 %>
  - task2: <%jmes expr2 %>

expr2 is evaluated only when task1 finishes. So expr2 can use any variable published by task1

5. Publish Complete

publish-on-complete element is processed in the last phase of the task. Here variables are published regardless of success or failure of the task's action. Its structure is similar to publish element.

publish-on-complete:
  - variable1: <%jmes expr1 %>
  - variable2: <%jmes expr2 %>

As with publish all the variables are published in order. So expr2 can use main.variable1 for its evaluation.

Iterative tasks

Tasks can be simple as described above but they can also be complex iterating over a list using the element with-items. The action phase of an iterative task behaves differently:

get_all_comments:
  with-items: <%jmes main.posts %>
  loop:
    task: get_comments
    input:
      postId: <%jmes item.id %>
    publish:
      combined: <%jmes {"post_details":item,"comments":task.comments} %>

For the task get_all_comments, action element is missing as it is mutually exclusive with with-items element. When the Action Phase of this task is executed, it will iterate over the list supplied by the expression <%jmes main.posts %> and in each iteration the current object is placed in the variable bucket item. loop element specifies what is to be done with each item. In this case task get_comments is called for each member of the list in with-items.

get_comments is not a function of an http client or a custom action. It is another task in the same workflow. This is where iterative task is different from simple tasks. Simple task calls an action in its action phase. An iterative task calls another task iteratively. Similarly input element here does not refer to the arguments for that action but a new main context for each task in the iteration. The main variable bucket available to the parent task is not automatically passed on to the child tasks (which are called iteratively). A new context is created for each iteration to keep logical separation. (All tasks in the loop are called concurrently).

Similarly, publish element of loop also behaves differently. Here we are publishing a list variable combined after the looped tasks are finished. And the expression <%jmes {"post_details":item,"comments":task.comments} %> specifies each element of the list which is published to combined variable rather than the contents of the data being published. The variable bucket task refers to the main bucket of each subtask after execution of the subtask is finished.

To recap, the task get_all_comments will iterate over each post in the list main.posts and call the task get_comments for each post with its own main bucket consisting of postId: <%jmes item.id %> and the final result of each subtask is published to variable combined which is a list with every element evaluated using expression <%jmes {"post_details":item,"comments":task.comments} %>. So comments of main.posts[0] will be available in main.combined[0] and so on. A full example can be read here.

While <task>/loop/publish behaves in this manner, <task>/publish is still available for simple publishing after the looping is done. Publish phase (and the rest of the subsequent phases) execute normally.

Similar to <task>/loop/publish we have <task>/loop/publish-on-error