Chapter 1: Introduction to Secret Rotation

Upon revisiting one of my self-hosted projects, I discovered that several of my GitHub tokens had expired. Initially, I thought it would be a simple fix—just update the secret value and redeploy the Docker Swarm stack with the Ansible playbook. However, I soon realized that managing Docker secrets is not as straightforward as it seems.

In Docker Swarm, once a secret is created, it is immutable, meaning it cannot be modified or deleted while in use. This limitation prompted me to devise a solution for efficiently rotating these secrets. Let's explore how to implement Docker Swarm secret rotation as an Ansible role, applicable to all playbooks that manage a stack.

Directory Structure

To create a reusable Ansible role, follow this directory structure:

rotate_docker_secrets/

├── tasks

│ ├── main.yml

├── defaults

│ └── main.yml

├── handlers

│ └── remove_dangling_secrets.yml

The completed code can be found at the end of this section.

Step 1: Calculate Checksums

To determine if a secret has changed, we calculate a checksum for each one. This allows us to track changes effectively.

• name: Calculate checksums for secrets

  set_fact:

  secret_checksums: "{{ secret_checksums | default({}) | combine({item.name: lookup('pipe', 'echo ' + item.value + ' | md5sum | cut -d" " -f1')}) }}"

  loop: "{{ secrets }}"

This ensures that any change in a secret results in a different checksum, guiding subsequent tasks.

Step 2: Create New Secrets

After calculating checksums, create new secrets in Docker Swarm. Each secret is tagged for easy identification during cleanup.

• name: Create new secrets

  docker_secret:

  name: "{{ item.name }}_{{ secret_checksums[item.name] }}"

  data: "{{ item.value }}"

  labels:

  managed_by: "rotate_docker_secrets"

  name: "{{ item.name }}"

  loop: "{{ secrets }}"

  register: new_secrets

Step 3: Update the Docker Compose File

Next, update the Docker Compose file to reference the new secrets. This method simplifies the role by performing straightforward search-and-replace actions.

• name: Replace secret names in Docker Compose file

  replace:

  path: "{{ docker_compose_path }}"

  regexp: "source: {{ item.name }}(_[a-f0-9]{32})?"

  replace: "source: {{ item.name }}_{{ secret_checksums[item.name] }}"

  loop: "{{ secrets }}"

Step 4: Clean Up Dangling Secrets

Finally, remove old, unused secrets to maintain a tidy environment. Identify secrets to keep and remove the rest.

• name: List all Docker secrets managed by this role

  command: docker secret ls --filter label=managed_by=rotate_docker_secrets --format "{{ '{{.Name}}' }}"

  register: existing_secrets

• name: Identify secrets to keep

  set_fact:

  secrets_to_keep: "{{ secrets_to_keep | default([]) + [item.name + '_' + secret_checksums[item.name]] }}"

  loop: "{{ secrets }}"

• name: Remove dangling secrets

  docker_secret:

  name: "{{ item }}"

  state: absent

  when: item not in secrets_to_keep

  loop: "{{ existing_secrets.stdout_lines }}"

  ignore_errors: yes

Step 5: Putting It All Together

Here is an example playbook that incorporates the rotate_docker_secrets role. Secrets are created by passing the list to the role, and the stack is redeployed once changes are made.

• hosts: localhost

  tasks:

  • name: Include rotate_docker_secrets role

    include_role:

    name: rotate_docker_secrets

    vars:

    secrets:

    • {name: 'caddy_github_client_id', value: "{{ lookup('env', 'CADDY_GITHUB_CLIENT_ID') }}"}
    • {name: 'caddy_github_client_secret', value: "{{ lookup('env', 'CADDY_GITHUB_CLIENT_SECRET') }}"}
    • {name: 'caddy_jwt_shared_key', value: "{{ lookup('env', 'CADDY_JWT_SHARED_KEY') | replace('\n', '\\n') }}"}
    • {name: 'caddy_digitalocean_api_token', value: "{{ lookup('env', 'CADDY_DIGITALOCEAN_API_TOKEN') }}"}

    checksum_directory: /path/to/checksums

• name: Deploy the updated Docker Compose configuration

  community.docker.docker_stack:

  state: present

  name: "{{ docker_stack_name }}"

  compose:

  • "{{ docker_compose_path }}"

  notify: clean up dangling secrets

Conclusion

Rotating secrets is vital for maintaining security. Given the immutable nature of Docker Swarm secrets, planning is essential. By utilizing Ansible, we can automate the entire process, ensuring efficient secret updates without disrupting services. This approach significantly simplifies secret management for Docker Swarm.

Consider integrating this Ansible role into your CI/CD pipeline to keep your secrets current. Ideally, secrets should auto-rotate, triggering the playbook to apply the changes.

If you're interested in executing Ansible playbooks with GitHub Actions, check out this Story:

Here are a few additional recommendations if you want to automate Ansible and GitHub Actions:

1. Automate Ansible Inventory creation with Terraform. Avoid manually setting up inventory and defining secrets in GitHub Actions.
2. Generate SSH keys with Terraform. Consider creating KNOWN_HOSTS and SSH_KEY directly with Terraform to eliminate configuration in the GitHub UI.

For more insights on DevOps, Kubernetes, and Security, follow me on Twitter and LinkedIn.