Posted on

Ansible 2.9 and AWS Dynamic Inventory Plugin

Like most other Devops folks using Ansible, I had been using the old dynamic inventory scripts for AWS located at:

 

https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.py

https://github.com/ansible/ansible/blob/devel/contrib/inventory/ec2.ini

I started using the dynamic inventory scripts back in the Ansible 1.9+ days. I completely missed the announcement in the 2.4 porting guide.  Some of the documentation still references the dynamic inventory – adding to my confusion.

Ansible 2.9 was released in November 2019, so I started the process of moving to it, using python 3, and moving to the inventory plugin framework.

I started off with a new ansible.cfg file.  Note the enable_plugins key in the [inventory]  section.

 

[defaults]
inventory = ./inventory
gather_timeout = 60
timeout = 60
interpreter_python =  auto_silent
callback_whitelist = yaml, debug, profile_tasks, timer
stdout_callback = yaml
bin_ansible_callbacks = True
transport = ssh
gathering = implicit
display_skipped_hosts = False
display_args_to_stdout = False
fact_caching = memory
#fact_caching_connection=localhost:6379:0
jinja2_extensions = jinja2.ext.loopcontrols

[ssh_connection]
ssh_args = -q -C -o ControlMaster=auto -o ControlPersist=600s
scp_if_ssh = True
pipelining = False

[inventory]
enable_plugins = constructed, aws_ec2, aws_rds
cache=True
cache_plugin=memory
#cache_connection=localhost:6379:0

Next, I had to configure the inventory plugin.  This took a fair amount of tweaking to make it backwards compatible.  However, the extra time in doing so allowed me to really simplify my group_vars.   Regions can be set to auto, and make sure to update the boto_profile with whatever you have in your ~/.aws/credentials file.

Also, I was able to create some shortcuts in the compose section. Specifically, the ansible_private_key_file is changed to the inventory_variable key_name, and then has the path ~/.ssh/ prepended to it.

ansible_host specifically tells ansible to use the private_ip_address variable.  You could change this to anything you want, such as public_ip_address.

hostnames: this is the order the dynamic inventory will use to set the {{ inventory_hostname }} variable.  I currently have it set to the Name tag, followed by the private IP.

keyed_groups is where I had to do some work to get my existing group_vars to match up without having to actually go through all of my accounts and inventory and update everything.

plugin: aws_ec2
boto_profile: my-aws-profile
regions:
  - us-west-1

filters:
  instance-state-name: ['running']
compose:
  ansible_host: private_ip_address
  ansible_private_key_file: key_name | regex_replace('(.*)', '~/.ssh/\1.pem')

hostnames:
  - tag:Name
  - private_ip_address 

include_extra_api_calls: yes
strict: yes
strict_permissions: yes

keyed_groups:
  - prefix: tag
    key: tags
  - prefix: aws
    key: instance_type
  - prefix: aws
    key: 'security_groups|json_query("[].group_id")'
  - prefix: "key"
    seperator: "_"
    key: key_name
  - prefix: "ami"
    seperator: ""
    key: image_id | regex_replace('ami.(.*)', '\1')
  - prefix: ""
    seperator: ""
    key: placement.region
  - prefix: platform
    key: 'platform | default("linux")'

That is the vast majority of what I had to change.   Now my existing groups such as:

inventory/group_vars/key_aws_keypair/ssh.yml and inventory/group_vars/platform_windows/connection.yml can still work.

 

And lastly, I went ahead and setup a new file in inventory/group_vars/all/account.yml.  In this file, I setup some basic mappings that I could have available to all of my AWS playbooks going forward.

 

---
_local_aws_inventory: "{{ lookup('file', 'inventory/aws_ec2.yml') | from_yaml }}"
_local_bastion: "{{ AWS_PROFILE | lower }}-linux"
_local_win_bastion: "{{ AWS_PROFILE | lower }}-windows"

AWS_DEFAULT_REGION: "{{ _local_aws_inventory.regions[0] }}"
AWS_PROFILE: "{{ _local_aws_inventory.boto_profile }}"
# Compatibility stuff
aws_default_region: "{{ AWS_DEFAULT_REGION }}"
aws_profile: "{{ AWS_PROFILE }}"
# Detect which cloud we are in
aws_cloud: "{{ 'aws-us-gov' if gov in aws_default_region else 'aws' }}"
aws_account_alias: "{{ AWS_PROFILE }}"

aws_connection_info: &aws_connection_info
  aws_region: "{{ aws_default_region }}"
  aws_profile: "{{ aws_profile }}"

ansible_user: ec2-user
ansible_ssh_common_args: "-o ProxyJump=ansible@{{ _local_bastion }}"

Now, I can use those in all of my playbooks going forward.  For example:

 

- hosts: all
  vars:
  roles:
  tasks:
    - name: Create s3 bucket
      s3_bucket:
        <<: *aws_connection_info
        bucket: mybucket
        ....

I hope this helps you migrate to the new Ansible dynamic inventory plugins. 

 

Thanks,

Brandon