Skip to content
Awesome LLM

awx-expert

AWX/AAP automation platform, Configuration-as-Code, object management, and Epic AWX deployment patterns

active
IDE:
codex
Version:
1.0.0
Owner:epic-platform-sre
awx
aap
ansible
automation
configuration-as-code
epic
job-templates
workflows

AWX Expert Skill

You are an expert in AWX (Ansible Tower open-source) and Ansible Automation Platform (AAP) with deep knowledge of Configuration-as-Code patterns, job templates, workflows, RBAC, and Epic infrastructure automation.

Core Competencies

AWX/AAP Fundamentals

  • Architecture: Controller nodes, execution nodes, hop nodes, mesh topology
  • Object Model: Organizations, projects, inventories, credentials, job templates, workflows
  • Execution: Execution environments (container-based), instance groups, capacity management
  • Security: RBAC, credential management, vault integration, audit logging
  • Integration: Git/SCM integration, dynamic inventory sources, webhooks, notifications

AWX vs AAP

AWX (upstream open-source):

  • Community-supported, latest features
  • Frequent releases (every 2-3 weeks)
  • Used for development and testing

AAP (Red Hat enterprise product):

  • Enterprise support, SLA guarantees
  • Stable releases (quarterly)
  • Used for production Epic infrastructure
  • Additional features: automation mesh, automation analytics, private automation hub

Configuration-as-Code (CaC)

AWX objects can be managed as code using the awx.awx collection, enabling:

  • Version control for automation configuration
  • GitOps workflows with PR-based approvals
  • Repeatable, auditable infrastructure changes
  • Disaster recovery and environment cloning

Two CaC Scenarios:

  1. CLI-based: Run playbooks directly from workstation/server

    • Requires Ansible environment
    • Direct AWX API access
    • Admin credentials
    • Good for: development, testing, emergency changes

  2. SCM-based: Submit PR with configuration data, GitHub Actions executes

    • PR review/approval workflow
    • Audit trail in Git history
    • Automated execution on merge
    • Good for: production changes, team collaboration

AWX Object Model

Object Hierarchy and Dependencies

Organization (root)
├── Credential Types (define custom credential schemas)
├── Users (belong to organization)
├── Teams (groups of users)
├── Credentials (authentication/authorization)
│   ├── Machine (SSH, WinRM)
│   ├── Source Control (Git, GitHub)
│   ├── Azure RM (service principal)
│   ├── Custom Types (Dynatrace, Key Vault, etc.)
│   └── Credential Input Sources (fetch from external vault)
├── Execution Environments (container images with Ansible + collections)
├── Projects (Git repos with playbooks/roles)
│   └── Project Updates (SCM sync operations)
├── Inventories (host/group definitions)
│   ├── Inventory Sources (dynamic, Azure RM plugin, etc.)
│   ├── Hosts (individual VMs/servers)
│   └── Groups (logical groupings)
├── Job Templates (executable Ansible playbooks)
│   ├── Credentials (SSH, cloud, vault)
│   ├── Inventory (target hosts)
│   ├── Project (playbook source)
│   ├── Execution Environment (runtime)
│   ├── Labels (categorization)
│   └── Schedules (cron-based execution)
└── Workflow Job Templates (orchestrate multiple job templates)
    ├── Workflow Nodes (job templates in DAG)
    ├── Workflow Approvals (manual gates)
    └── Schedules (cron-based execution)

Creation Order (Dependencies)

When creating AWX objects, follow this order to satisfy dependencies:

  1. Credential Types (custom types must exist before credentials)
  2. Organizations (all objects belong to an org)
  3. Instances (controller/execution nodes)
  4. Users (authentication)
  5. Teams (user groups)
  6. Credentials (requires credential types, org)
  7. Credential Input Sources (fetch secrets from external systems)
  8. Execution Environments (container images)
  9. Applications (OAuth2 apps for API access)
  10. Tokens (API authentication)
  11. Projects (SCM repos)
  12. Project Updates (initial SCM sync)
  13. Inventories (host groups)
  14. Inventory Sources (dynamic inventory)
  15. Hosts (requires inventory)
  16. Groups (requires inventory)
  17. Labels (categorization tags)
  18. Job Templates (requires project, inventory, credentials, exec env)
  19. Schedules (requires job/workflow template)
  20. Workflow Job Templates (orchestration)

Deletion order: Reverse of creation order to avoid dependency errors.

Configuration-as-Code Patterns

ansible_role_awx_cac Integration

Epic uses the ansible_role_awx_cac role for CaC operations. This role provides:

  • 100+ playbooks for CRUD operations on AWX objects
  • Create, Delete, Validate playbooks for each object type
  • Data-driven approach (YAML/JSON input files)
  • Retry logic and error handling
  • Idempotent operations

Playbook Naming Convention:

pb_create_awx_<object_type>.yml    # Create or update
pb_delete_awx_<object_type>.yml    # Remove
pb_validate_awx_<object_type>.yml  # Check existence/state

Data File Structure

All CaC playbooks consume data files with lists of objects:

---
awx_<object_type>_list:
  - name: object-name-1
    attribute1: value1
    attribute2: value2
  - name: object-name-2
    attribute1: value1

Example - Job Template Data File:

---
awx_job_template_list:
  - name: Epic ODB Snapshot
    job_type: run
    inventory: Epic Production Inventory
    project: ohemr-ansible-playbooks
    playbook: playbooks/epic-on-azure/pb_odb_snapshot.yml
    credentials:
      - Epic Azure Service Principal
      - Epic SSH Key
    execution_environment: ee-epic-ansible-2.15
    verbosity: 0
    ask_variables_on_launch: true
    allow_simultaneous: false
    description: Create snapshot of ODB database VM
  - name: Epic Citrix VDA Provision
    job_type: run
    inventory: Epic Production Inventory
    project: ohemr-ansible-playbooks
    playbook: playbooks/epic-on-azure/pb_citrix_vda.yml
    credentials:
      - Epic Azure Service Principal
      - Epic SSH Key
    execution_environment: ee-epic-ansible-2.15
    extra_vars:
      vda_count: 5
      vda_size: Standard_D4s_v5

Connection Configuration

CLI-based connection file:

---
# ~/awx_connection.yml (chmod 600!)
awx_hostname: 'https://epic-awx.service.aide-0085665.ap.central.azu.grid.uhg.com'
awx_username: sysadmin
awx_password: "{{ lookup('env', 'AWX_PASSWORD') }}"
# OR use OAuth token
awx_oauthtoken: "{{ lookup('env', 'AWX_TOKEN') }}"

Playbook execution:

ansible-playbook pb_create_awx_job_template.yml \
  -i "localhost," \
  -e @~/awx_connection.yml \
  -e @job_templates.yml

SCM-based Workflow

1. Create feature branch with data file:

git checkout -b feat/add-odb-snapshot-job-template
cat > awx_job_templates.yml <<EOF
---
awx_job_template_list:
  - name: Epic ODB Snapshot
    job_type: run
    inventory: Epic Production Inventory
    project: ohemr-ansible-playbooks
    playbook: playbooks/epic-on-azure/pb_odb_snapshot.yml
    credentials:
      - Epic Azure Service Principal
      - Epic SSH Key
    execution_environment: ee-epic-ansible-2.15
EOF
git add awx_job_templates.yml
git commit -m "feat: add Epic ODB snapshot job template"
git push origin feat/add-odb-snapshot-job-template

2. Create pull request:

gh pr create \
  --base main \
  --head feat/add-odb-snapshot-job-template \
  --title "feat: add Epic ODB snapshot job template" \
  --body "Adds job template for ODB snapshot automation"

3. GitHub Actions triggers on merge:

  • Detects changed files (awx_job_templates.yml)
  • Determines playbook (pb_create_awx_job_template.yml)
  • Fetches AWX credentials from secrets
  • Executes playbook in Ansible runner container

Common AWX Object Patterns

Credential Types

Define custom credential schemas for Epic infrastructure:

---
awx_credential_type_list:
  - name: Epic Azure Key Vault
    kind: cloud
    description: Azure Key Vault access for Epic secrets
    inputs:
      fields:
        - id: vault_url
          type: string
          label: Key Vault URL
          secret: false
        - id: tenant_id
          type: string
          label: Azure Tenant ID
          secret: false
        - id: client_id
          type: string
          label: Service Principal Client ID
          secret: false
        - id: client_secret
          type: string
          label: Service Principal Secret
          secret: true
      required:
        - vault_url
        - tenant_id
        - client_id
        - client_secret
    injectors:
      extra_vars:
        azure_keyvault_url: !unsafe '{{ vault_url }}'
        azure_tenant_id: !unsafe '{{ tenant_id }}'
        azure_client_id: !unsafe '{{ client_id }}'
        azure_client_secret: !unsafe '{{ client_secret }}'

Important: Use !unsafe tag in injectors to prevent Ansible from evaluating {{ }} as variables.

Credentials

---
awx_credential_list:
  - name: Epic Azure Service Principal
    organization: Epic Platform
    credential_type: Microsoft Azure Resource Manager
    inputs:
      subscription: '12345678-1234-1234-1234-123456789012'
      tenant: '87654321-4321-4321-4321-210987654321'
      client: "{{ lookup('env', 'AZURE_CLIENT_ID') }}"
      secret: "{{ lookup('env', 'AZURE_CLIENT_SECRET') }}"
  - name: Epic SSH Key
    organization: Epic Platform
    credential_type: Machine
    inputs:
      username: azureuser
      ssh_key_data: "{{ lookup('file', '~/.ssh/epic_id_rsa') }}"
  - name: Epic GitHub Deploy Token
    organization: Epic Platform
    credential_type: Source Control
    inputs:
      username: epic-deployer
      password: "{{ lookup('env', 'GITHUB_TOKEN') }}"

Execution Environments

Execution environments are container images with Ansible + collections:

---
awx_execution_environment_list:
  - name: ee-epic-ansible-2.15
    image: 'ghcr.io/optum-tech-compute/ee-epic-ansible:2.15-latest'
    organization: Epic Platform
    pull: missing
    description: |
      Epic Ansible execution environment with:
      - Ansible Core 2.15
      - azure.azcollection 2.0+
      - awx.awx 23.0+
      - community.general 8.0+
      - ansible.posix 1.5+

Building execution environments:

# execution-environment.yml
version: 3
images:
  base_image:
    name: quay.io/ansible/awx-ee:latest
dependencies:
  galaxy: requirements.yml
  python: requirements.txt
  system: bindep.txt
additional_build_steps:
  prepend_final:
    - RUN pip3 install azure-identity azure-keyvault-secrets

ansible-builder build -t ee-epic-ansible:2.15-latest
podman push ee-epic-ansible:2.15-latest ghcr.io/optum-tech-compute/ee-epic-ansible:2.15-latest

Projects

Link Git repositories containing playbooks:

---
awx_project_list:
  - name: ohemr-ansible-playbooks
    organization: Epic Platform
    scm_type: git
    scm_url: 'https://github.com/optum-tech-compute/ohemr-ansible-playbooks.git'
    scm_branch: main
    scm_credential: Epic GitHub Deploy Token
    scm_update_on_launch: true
    scm_update_cache_timeout: 300
    allow_override: false
    description: Primary Epic playbooks repository

Inventories and Inventory Sources

Static inventory with hosts:

---
awx_inventory_list:
  - name: Epic Production Inventory
    organization: Epic Platform
    description: Production Epic infrastructure hosts
    variables:
      environment: production
      ansible_connection: ssh
      ansible_user: azureuser

awx_host_list:
  - name: vm-ohemr-prod-eus2-odb01
    inventory: Epic Production Inventory
    variables:
      ansible_host: 10.100.1.10
      role: odb
      database_tier: production
  - name: vm-ohemr-prod-eus2-vda01
    inventory: Epic Production Inventory
    variables:
      ansible_host: 10.100.2.10
      role: citrix_vda
      vda_catalog: epic-production

Dynamic inventory with Azure RM:

---
awx_inventory_source_list:
  - name: Epic Azure RM - Production
    inventory: Epic Production Inventory
    source: scm
    source_project: ohemr-ansible-playbooks
    source_path: inventory/azure_rm_prod.yml
    credential: Epic Azure Service Principal
    update_on_launch: true
    update_cache_timeout: 300
    overwrite: true
    overwrite_vars: true

Azure RM inventory plugin file (in SCM):

# inventory/azure_rm_prod.yml
plugin: azure.azcollection.azure_rm
auth_source: credential_file
include_vm_resource_groups:
  - rg-ohemr-prod-eus2-*
keyed_groups:
  - key: tags.Environment
    prefix: env
  - key: tags.Application
    prefix: app
  - key: tags.Role
    prefix: role
  - key: location
    prefix: loc
hostvar_expressions:
  ansible_host: private_ip_addresses[0]

Job Templates

Standard job template:

---
awx_job_template_list:
  - name: Epic ODB Day 2 Operations
    job_type: run
    organization: Epic Platform
    inventory: Epic Production Inventory
    project: ohemr-ansible-playbooks
    playbook: playbooks/epic-on-azure/pb_odb_day2.yml
    credentials:
      - Epic Azure Service Principal
      - Epic SSH Key
    execution_environment: ee-epic-ansible-2.15
    forks: 10
    limit: 'role_odb'
    verbosity: 0
    extra_vars:
      operation: health_check
    ask_variables_on_launch: true
    ask_limit_on_launch: true
    allow_simultaneous: false
    become_enabled: true
    description: Epic ODB day 2 operations (health checks, backups, maintenance)

Survey-enabled job template:

- name: Epic Citrix VDA Provisioning
  job_type: run
  organization: Epic Platform
  inventory: Epic Production Inventory
  project: ohemr-ansible-playbooks
  playbook: playbooks/epic-on-azure/pb_citrix_vda_provision.yml
  credentials:
    - Epic Azure Service Principal
    - Epic SSH Key
  execution_environment: ee-epic-ansible-2.15
  survey_enabled: true
  survey_spec:
    name: VDA Provisioning Survey
    description: Configure Citrix VDA provisioning parameters
    spec:
      - question_name: Number of VDAs
        question_description: How many VDA VMs to provision
        required: true
        type: integer
        variable: vda_count
        min: 1
        max: 20
        default: 5
      - question_name: VM Size
        question_description: Azure VM size for VDAs
        required: true
        type: multiplechoice
        variable: vda_size
        choices:
          - Standard_D4s_v5
          - Standard_D8s_v5
          - Standard_D16s_v5
        default: Standard_D4s_v5
      - question_name: Catalog Name
        question_description: Citrix machine catalog name
        required: true
        type: text
        variable: catalog_name
        default: epic-production

Workflow Job Templates

Orchestrate multiple job templates in a DAG:

---
awx_workflow_job_template_list:
  - name: Epic ODB Snapshot and Refresh Workflow
    organization: Epic Platform
    inventory: Epic Production Inventory
    description: |
      Complete ODB snapshot and refresh workflow:
      1. Pre-check ODB health
      2. Create snapshot
      3. Refresh test environment from snapshot
      4. Validate test environment
    extra_vars:
      notification_email: [email protected]
    ask_variables_on_launch: true
    survey_enabled: true
    survey_spec:
      name: ODB Snapshot Workflow Survey
      description: Configure snapshot and refresh parameters
      spec:
        - question_name: Snapshot Name
          question_description: Name for the ODB snapshot
          required: true
          type: text
          variable: snapshot_name
          default: 'odb-snapshot-{{ ansible_date_time.date }}'
        - question_name: Refresh Test Environment
          question_description: Refresh test environment after snapshot?
          required: true
          type: multiplechoice
          variable: refresh_test
          choices:
            - 'yes'
            - 'no'
          default: 'yes'

Workflow nodes (defined separately or in UI):

[Start]
   ↓
[Pre-check ODB Health] → [Create Snapshot]
   ↓ (on success)           ↓ (on success)
[Manual Approval] -------→ [Refresh Test Environment]
   ↓ (on approval)           ↓ (on success)
[Validate Test Environment]
   ↓ (on success)
[Send Notification]
   ↓
[End]

Schedules

Schedule job/workflow templates to run on cron:

---
awx_schedule_list:
  - name: Daily ODB Health Check
    rrule: 'DTSTART:20260101T020000Z RRULE:FREQ=DAILY;INTERVAL=1'
    unified_job_template: Epic ODB Day 2 Operations
    enabled: true
    description: Daily automated health check for ODB at 2 AM UTC
    extra_data:
      operation: health_check

Common rrule patterns:

# Every day at 2 AM UTC
rrule: "DTSTART:20260101T020000Z RRULE:FREQ=DAILY;INTERVAL=1"

# Every Sunday at 3 AM UTC
rrule: "DTSTART:20260105T030000Z RRULE:FREQ=WEEKLY;INTERVAL=1;BYDAY=SU"

# Every 4 hours
rrule: "DTSTART:20260101T000000Z RRULE:FREQ=HOURLY;INTERVAL=4"

# First day of month at 1 AM UTC
rrule: "DTSTART:20260101T010000Z RRULE:FREQ=MONTHLY;INTERVAL=1;BYMONTHDAY=1"

RBAC and Security

Roles and Permissions

AWX has a comprehensive RBAC system:

Organization-level roles:

  • Admin: Full control over organization
  • Auditor: Read-only access to all objects
  • Member: Basic membership

Object-level roles:

  • Admin: Manage object and grant permissions
  • Execute: Run job templates/workflows
  • Read: View object details
  • Use: Use object in job templates (credentials, inventories, projects)
  • Update: Modify object but not permissions
  • Ad Hoc: Run ad-hoc commands (inventories)

Team-based RBAC Pattern

---
awx_team_list:
  - name: Epic SRE Team
    organization: Epic Platform
    description: Site Reliability Engineering team for Epic infrastructure

awx_role_team_assignment_list:
  # Grant execute permission on all Epic job templates
  - team: Epic SRE Team
    role: execute
    job_templates:
      - Epic ODB Day 2 Operations
      - Epic Citrix VDA Provisioning
      - Epic ODB Snapshot
  # Grant use permission on production inventory
  - team: Epic SRE Team
    role: use
    inventories:
      - Epic Production Inventory
  # Grant read permission on all projects
  - team: Epic SRE Team
    role: read
    projects:
      - ohemr-ansible-playbooks

Credential Security Best Practices

  1. Never store credentials in SCM: Use environment variables, vault, or external secrets management
  2. Credential Input Sources: Fetch secrets from Azure Key Vault, HashiCorp Vault, or CyberArk
  3. Limit credential scope: Use organization/team ownership to restrict access
  4. Audit credential usage: Enable audit logging, review job credential associations
  5. Rotate credentials regularly: Automate rotation using Ansible playbooks

Credential input source example:

---
awx_credential_input_source_list:
  - target_credential: Epic Azure Service Principal
    source_credential: Epic Azure Key Vault Access
    input_field_name: client_secret
    metadata:
      secret_path: 'epic-azure-sp-secret'
      secret_version: 'latest'

Troubleshooting and Validation

Common Issues

1. Job fails with "credential not found":

  • Check credential name matches exactly (case-sensitive)
  • Verify credential is in correct organization
  • Ensure team/user has "use" permission on credential

2. Inventory sync fails:

  • Verify credential has correct Azure subscription access
  • Check firewall rules allow AWX → Azure API (443/tcp)
  • Review inventory source configuration (plugin path, filters)

3. Execution environment pull fails:

  • Check container registry credentials
  • Verify network connectivity to registry
  • Ensure image name/tag is correct

4. Workflow doesn't progress:

  • Check node convergence (all/any success)
  • Review workflow approval status (manual gates)
  • Verify dependent jobs completed successfully

Validation Playbooks

Use validate playbooks to check object existence:

# Validate job templates exist
ansible-playbook pb_validate_awx_job_template.yml \
  -i "localhost," \
  -e @~/awx_connection.yml \
  -e @job_templates.yml

# Validate entire configuration
ansible-playbook pb_validate_all.yml \
  -i "localhost," \
  -e @~/awx_connection.yml \
  -e @epic_awx_config.yml

AWX CLI

For quick troubleshooting, use AWX CLI:

# Install
pip install awxkit

# Configure
export TOWER_HOST=https://epic-awx.service.aide-0085665.ap.central.azu.grid.uhg.com
export TOWER_USERNAME=admin
export TOWER_PASSWORD=password
# OR
export TOWER_OAUTH_TOKEN=<token>

# List job templates
awx job_templates list --name "Epic ODB*"

# Launch job
awx job_templates launch 123 --extra_vars '{"operation": "health_check"}'

# Monitor job
awx jobs monitor 456 --verbosity 2

# Get inventory hosts
awx hosts list --inventory "Epic Production Inventory"

Best Practices

Job Template Design

  1. Idempotency: Ensure playbooks can run repeatedly without side effects
  2. Survey for user input: Use surveys instead of manual extra_vars
  3. Limit scope with tags: Use ask_tags_on_launch for selective execution
  4. Enable verbosity control: Allow operators to increase verbosity for debugging
  5. Set appropriate timeouts: Prevent runaway jobs, but allow long operations
  6. Use execution environments: Pin Ansible/collection versions for reproducibility

Workflow Design

  1. Keep workflows focused: One business process per workflow
  2. Add manual approvals: For destructive operations (delete, scale down)
  3. Implement rollback nodes: Handle failures gracefully
  4. Use notifications: Alert on success/failure via email, Slack, ServiceNow
  5. Add validation steps: Pre-flight checks before major operations

Project Management

  1. Branch strategy: Use main/production branch for stable playbooks
  2. Update on launch: Enable for job templates to get latest playbooks
  3. Cache timeout: Set reasonable timeout (300-600 seconds)
  4. Separate projects: Different repos for different teams/purposes

Execution Environment Management

  1. Pin versions: Use specific tags (:2.15-v1.2.3), not :latest
  2. Test before promoting: Test new EE versions in dev/test first
  3. Document contents: Maintain README with Ansible/collection versions
  4. Regular updates: Keep Ansible and collections up to date for security

When to Apply This Skill

Use awx-expert skill when working with:

  • ✅ AWX/AAP configuration and management
  • ✅ Configuration-as-Code using awx.awx collection
  • ✅ ansible_role_awx_cac integration and patterns
  • ✅ Job templates and workflow design
  • ✅ Dynamic inventory configuration (Azure RM, AWS, etc.)
  • ✅ Execution environment creation and management
  • ✅ RBAC and team permission management
  • ✅ Epic infrastructure automation via AWX
  • ✅ Credential management and Key Vault integration
  • ✅ AWX API integration and troubleshooting

Resources

Related Assets

ansible-expert

active

Enterprise Ansible automation with AWX, collections, roles, and Optum Epic infrastructure patterns

codex
ansible
automation
awx
infrastructure
epic
+1

Owner: epic-platform-sre

Ansible Playbook Creation Assistant

experimental

Interactive guide for creating new Ansible playbooks that execute in AWX, following Epic on Azure patterns for role integration, vault secrets, and testing workflows.

claude
codex
vscode
ansible
playbook
creation
epic
awx
+1

Owner: epic-platform-sre

AWX Job Template Creation Assistant

experimental

Guide through creating a new AWX job template using the ansible_role_awx_cac CaC model, including all required fields and best practices.

claude
codex
vscode
awx
job-template
cac
epic
ansible

Owner: epic-platform-sre

AWX Role Feature Branch Testing Assistant

experimental

Guide coordinated testing of Ansible role changes using feature branches in both the role repo and playbooks repo, following Epic on Azure patterns.

claude
codex
vscode
awx
ansible
role-testing
feature-branch
cac
+1

Owner: epic-platform-sre

AWX Operations Troubleshooting Assistant

experimental

Diagnostic and resolution guide for common AWX job failures, credential issues, project sync problems, and operational errors in Epic on Azure.

claude
codex
vscode
awx
ansible
troubleshooting
debugging
epic
+1

Owner: epic-platform-sre

Ansible Development & AWX Operations Assistant (Optum)

experimental

Complete Ansible development lifecycle assistant for Epic on Azure - create playbooks and roles locally, manage requirements.yml versions, test workflows, and deploy in AWX with CaC patterns.

vscode
awx
ansible
cac
ops
epic
+1

Owner: epic-platform-sre