Ansible collection processing

2019-04-25 ansible

As a Java developer, I sometimes dream that I can use the Java 8+ Stream API in my Ansible playbook to process list and dict variables.

In this article, I’ll show you can process a list of users:

users:
  - id: bouh
    name: "Mary"
    admin: True
    role: child
  - id: sulli
    name: "James Sullivan"
    admin: False
    role: monster
  - id: bob
    name: "Bob Wazowski"
    admin: False
    role: assistant
  - id: celia
    name: "Celia Mae"
    admin: False
    role: assistant

Jinja Filters

The main tool to transform Ansible variables are Jinja filters. There are 2 libraries of filters available:

Filters are similar to Unix or Anguler pipes and can be chained.

Like in other data processing libraries there two kinds of operators:

  • Mappers take stream of element and produce a stream of elements: selectattr, rejectattr, map, list

  • Reducers: take a stream of elements and produce a single element: join, first, last, max, min

admin_user_ids: |
  {{ users
  |selectattr('admin')
  |map(attribute='id')
  |join(',') }} (1)
normal_user_count: |
  {{ users
  |rejectattr('admin')
  |list |count }} (2)
1 Take the id attribute of users having admin set to true and join them.
2 Take the users havin admin set to false and count them. As the rejectattr filter returns an iterator, but the count filter requires a list, I have to use list filter to convert it.

The selectattr/rejectattr filters can take 3 arguments: the attribute, a boolean operator and an argument. The operator can be chosen among Jinja’s builtin tests. This list can also be found in Ansible source code tests.py.

assistant_user_ids: |
  {{ users
  |selectattr('role', 'equalto', 'assistant')
  |map(attribute='id')
  |join(',') }}

With Ansible 2.7+, the map filter can take 3 arguments: the attribute, an operator and arguments. The operator can be chosen among Jinja filters, and will be applied to each element of the list.

user_first_names: |
  {{ users
  |map(attribute='name')
  |map('regex_replace', '(\\w+)( .*)?', '\\g<1>')
  |join(',') }} (1)
1 For each users take its name and when the regular expression matches apply the replacement, then join the result.

JSON Query

Another strategy is to use a JSON Path to walk down the YAML tree. It’s bit less verbose and bit more powerful than the previous solution.

jq_admin_user_ids: |
  {{ users
  |json_query("[?admin].id")
  |join(',') }} (1)
jq_assistant_user_ids: |
  {{ users
  |json_query("[?role == 'assistant'].id")
  |join(',') }} (2)
1 Take the id attributes of users having admin set to true and then join them.
2 Take the id attributes of users having role set to assistant and then join them.

This json_query is based on the jmespath Python library, this means 2 things:

  1. You can use jmespath.org web site to cook your JSON path query.

  2. You’ll have to add the jmespath library to your Python environnement.

Sadly, nesting JMESPath expressions inside Jinja template expressions inside YAML files can be tricky. This following example fails, even if the JMES path is alright in Python interpreter.

jq_bid_user_ids: |
  {{ users
  |json_query("[?starts_with(id,'b')].id")
  |join(',') }}

Conclusion

It’s possible to transform a variable containing an array into another list. However it’s still painful to do because neither YAML nor Jinja tare programming languages. I personnaly regret I can’t invoke Python code from Ansible playbook and use for comprehensions, imagine something like:

py_admin_user_ids: |
  {{ ','.join([ user.id for user in users if user.admin ]) }}

Other posts