How to convert a config file (of lines) to a dict list in Ansible

Question

Convert from many text lines to a list of dictionaries items.

I'm trying to convert a configuration file to a list of dict entries. That list can by used by Ansible for various tasks. The file is a general text .conf file with "item=value" pairs. I need to convert the file to a list of dicts. variable => [ "item1":"value", "item2":"value", ... ]

I followed this solution: Converting output of Python script to dict in Ansible Got a partial solution as I only got the last line from the file and only one dict in the variable. (A single dict list)

A part of the .conf file

# Flags if dhcp should be used or not
use_dhcp=1

# The NOS300s IP address
ip_address=

# Netmask to use for this NOS300
netmask=

# Default gateway
gateway=

# DNS Server
dns_server=

# Local folder to download the files to (required, absolute path)
# download_folder = /tmp

All of the comments and blanks need to be removed and then convert the lines with '=' into a list of dicts.

<< playbook def >>
  vars:
     remote_config_file: /etc/my/general.conf

  tasks:
  - name: "Get {{ remote_config_file }} to a list of terms and add to var"
    shell: "egrep -v \'(^$|^#)\' {{remote_config_file}} | grep \'=\' "
    register: NosConf

  - name: "Convert to dict list"
    set_fact:
      NosFactDict: "{{ parameter | default({}) | combine ( { item.split('=')[0]: item.split('=')[1] } ) }}"
    with_items: "{{ NosConf.stdout_lines }}"

  - debug:
      msg: "{{ NosFactDict }}"

The shell command strips the blank and commented lines then filters the lines with '=' then split and combine the filtered line to a dict.

I want a list longer then:

task path: /media/sf_Virtualbox-share/FactfileAdd.yml:30
ok: [192.168.2.112] => {
    "msg": {
        "docker0_network": "172.17.0.0/16"
    }
}

It need to be more then

ok: [192.168.2.112] => {
    "msg": {
        "base_path": "/var/www/desert", 
        "buildnumber": "os", 
        ...
        "use_proxy": "0", 
        "use_sat_distribution": "0", 
        "version": "1.6.1.8-os"
    }
}

`NosConf` won't keep all data if there are [multiline values](https://docs.python.org/3/library/configparser.html#supported-ini-file-structure). — Vladimir Botka, Aug 20 '19 at 12:54

score 1 · Accepted Answer · answered Aug 20 '19 at 10:00

Your are doing great, but the prat you miss is about concatenation of the dict into a list. This can be done like this:

   - name: "Convert to dict list"
    set_fact:
      NosFactDict: "{{ NosFactDict|default([]) + [ parameter | default({}) | combine ( { item.split('=')[0]: item.split('=')[1] } ) ] }}"
    with_items: "{{ NosConf.stdout_lines }}"

So you concatenate your NosFactDict with + [ <your_item> ]. And the |default([]) is there to declare NosFactDict as an empty list during the first loop (to avoid an error.

Ref: Similar queston on StackOverlow

score 1 · Answer 2 · answered Aug 20 '19 at 12:11

I did find a working solution similar to xenlo answer, but set up the output var in two steps.

  - name: Set default dict for NosFactDict
    set_fact:
      NosFactDict: {}

  - name: "Convert to dict list"
    set_fact:
      NosFactDict: "{{ NosFactDict | combine ( { item.split('=')[0]: item.split('=')[1] } ) }}"
    with_items: "{{ NosConf.stdout_lines }}"

I suspect I could find a version with "default" that works. This is a clunkier solution, yet it does work. If I am performing more tasks like this then I will run xenlo's solution as that looks much faster to run.

score 0 · Answer 3 · answered Aug 20 '19 at 11:54

The tasks below do the job, I think.

- name: Create list of variables
  set_fact:
    my_vars: "{{ my_vars|default([]) +
                 [item.split('=').0|trim] }}"
  with_lines: "cat {{ remote_config_file }}"
  when: item is match('^(\s*[a-zA-Z0-9_]+\s*)=(.*)$')

- name: Create list of values
  set_fact:
    my_values: "{{ my_values|default([]) +
                   [lookup('ini', item ~ ' type=properties file=' ~ remote_config_file)] }}"
  loop: "{{ my_vars }}"

- name: Create dictionary of variables and values
  set_fact:
    my_dict: "{{ my_dict|default({})| 
                 combine({item.0: item.1}) }}"
  with_together:
    - "{{ my_vars }}"
    - "{{ my_values }}"

- debug:
    var: my_dict

Note. Unfortunately ini plugin returns only values. (see source)

if is_regexp:
    return [v for k, v in self.cp.items(section) if re.match(key, k)]

How to convert a config file (of lines) to a dict list in Ansible

Convert from many text lines to a list of dictionaries items.

3 Answers3