Backup structure

Backups are bundled into tarballs containing the mongo, Git, keycloak, vault, and model secrets, as well as a config file for the backup.

The bundles use the filename convention YYMMDD-HHMMSS.tar.gz, and all content in the tarball is in a directory with the same timestamp format. If you created a backup on December 16th, 2022 at 8:57:10 pm, the tarball 20221216-205710.tar.gz contains the following:

  • 20221216-205710/

    • 20221216-205710/config.yaml (Config file for backup)

    • 20221216-205710/mongo_archive_local-backup.gz (Mongo archive dump of primary Domino application database)

    • 20221216-205710/git.tar.gz (tarball of git repos and critical user project metadata/version control)

    • 20221216-205710/keycloak-postgres_archive_local-backup.gz (Postgres database dump)

    • 20221216-205710/vault-postgres_archive_local-backup.gz (Postgres database dump)

    • 20221216-205710/vault-k8s-secrets_local-backup.yaml (Kubernetes secrets and critical secrets to unlock Vault)

    • 20221216-205710/model_secrets-k8s-secrets_local-backup.yaml (Kubernetes secrets and Environment variables for models)

Caution

The backup bundle only contains data from the Kubernetes cluster. This process does not back up data that is stored in object storage (for example, AWS S3 buckets, Azure containers, GCP buckets) on shared storage like NFS or CIFS/SMB or in an external Docker Registry. Typically, the infrastructure management function operates object storage externally. This is reliable and often large. The infrastructure admin is responsible for the backup/restore strategy for this data.