Scheduled backup of Vault secrets with CronJob of Kubernetes

I am a DevOps engineer at and I will share in this post my experience related to automation of Vault backup creation using Kubernetes CronJob. 

This post is a continuation of the previous post:

The repository with all the code:

What is HashiCorp’s Vault?

Vault is a tool for securely accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, certificates, and more. Vault provides a unified interface to any secret while providing tight access control and recording a detailed audit log.

My Setup

  • EKS Kubernetes cluster
  • Vault runs on EKS cluster

What you will learn from this post?

  • How to create a scheduled backup for Vault secrets with CronJob of Kubernetes.

  • How to add Prometheus alerts for failed jobs.

You can find all the code presented in my repository:

Let’s start.

Building the docker container

First, need to build a docker container based on python3 and include the code of

Need clone the repo first with Docker file: ‘git clone’

Docker file:

FROM python:3

COPY requirements.txt /

RUN pip install -r requirements.txt


CMD [ "python", "./" ]

Building image

# login to dockerhub
$ docker login -u YOUR_USERNAME -p YOUR_PASSWORD

# Build Docker
$ docker build -t vault-backup .

Validate docker container working properly

$ docker run --name test-vault-backup --rm vault-backup
Specify one of the commands below

It’s working, we got a list of commands from vault-backup:-)

Pushing vault-backup docker container to docker hub

$ docker tag vault-backup <Your Docker ID>/vault-backup:latest
$ docker push <Your Docker ID>/vault-backup:latest

In my case, it’s ‘warolv/vault-backup:latest’, you can find an already built image there.

CronJob to run vault-backup on a daily basis

apiVersion: batch/v1beta1
kind: CronJob
  name: vault-backup
  schedule: "0 1 * * *"
          restartPolicy: Never
            instance-type: spot
            - name: awscli
              image: amazon/aws-cli:latest
                - "aws"
                - "s3"
                - "cp"
                - "/data/vault_secrets.enc"
                - "s3://jenkins-backups/vault_secrets.enc"
              imagePullPolicy: Always
                - secretRef:
                    name: aws-creds-secret
              - name: backup-dir
                mountPath: /data
            - name: vault-backup
              image: warolv/vault-backup
                - "python3"
                - ""
                - "dump"
                - "-dp"
                - "/data/vault_secrets.enc"
              imagePullPolicy: Always
                - secretRef:
                    name: vault-backup-secret
              - name: backup-dir
                mountPath: /data
          - name: backup-dir
            emptyDir: {}


  • First ‘’ script will run from InitContainer and secrets dump will be created (vault_secrets.enc) and saved to /data folder which is a shared folder for both containers.

  • The second will run ‘awscli’ container which will be used to push the secrets dump to a private S3 bucket (AWS CLI is used to copy the secrets dump to the privare S3 bucket). Of course, S3 private bucket must exist.

  • Credentials for AWS CLI (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY) and for vault_backup script exported to the environment as k8s secrets.

  • In this example, I am copying the dump to ‘s3://jenkins-backups/vault_secrets.enc’, in the production use case I suggest adding a timestamp to dump of secrets to be something like vault_secrets_${timestamp}.enc

Creating secrets for CronJob

# create k8s secret for AWS
$ kubectl create secret generic aws-creds-secret \

# create k8s secret with all needed data for vault-backup
$ kubectl create secret generic vault-backup-secret \
--from-literal=VAULT_ADDR=http://vault.vault.svc.cluster.local:8200 \
--from-literal=ROLE_ID=YOUR_ROLE_ID \
--from-literal=SECRET_ID=YOUR_SECRET_ID \

It’s only an example, you need to put real values.

Deploy vault-backup cronjob

$ kubectl apply -f examples/cronjob/cronjob.yaml

How to trigger a Job from CronJob?

In case you want to test your job is working properly:

$ kubectl create job --from=cronjob/vault-backup vault-backup-001

Adding alerts to Prometheus

I am using kube-state-metrics with Prometheus and we have these metrics available:

Let’s add an alert for ‘failed job’ and for cronjob which ‘takes too much time’, of course, it’s only an example to give you an idea.

- name: cronjob.rules
  - alert: SlowCronJob
    expr: time()-kube_cronjob_next_schedule_time > 1800
    for: 30m
      severity: warning
      description: CronJob / is taking more than 30m to complete
      summary: CronJob taking more than 30m
  - alert: FailedJob
    expr: kube_job_status_failed  > 0
    for: 30m
      severity: warning
      description: Job / failed
      summary: Job failure

In this post, I described how to automate Vault backup creation using Kubernetes CronJob and a simple python script that I built.

Thank you for reading, I hope you enjoyed it, see you in the next post.

If you want to be notified when the next post of this tutorial is published, please follow me on Twitter @warolv.

For consulting gigs you can reach me on Upwork

Instagram: @warolv

Medium account:




Leave a comment