Best practice for production deployments
This document guides you through the process of setting up a production-ready Azimuth deployment following recommended best practice.
Repository
Before building your Azimuth configuration, you must first
set up your configuration repository, including initialising
git-crypt for encrypting secrets.
It is recommended to use a feature branch workflow to make changes to your Azimuth configuration in a controlled way.
Environments
An Azimuth configuration repository contains multiple environments, some of which contain common configuration ("mixin" environments), and some of which represent a deployment ("concrete" environments).
For a production deployment of Azimuth, there should be at least two concrete environments
that are deployed from the main branch:
production- the Azimuth deployment that is provided to end usersstaging- used to validate changes before pushing to production
Both of these environments should be highly-available deployments, in order to fully test the upgrade process.
It is recommended to have a
single node aio environment that
also follows the production environment as closely as possible. This can be used for
testing changes to the configuration before they are merged to main.
In order for validation in the aio and staging environments to be meaningful, the
configuration of these environments should be as similar to production as possible.
To do this, two site-specific mixins should be used:
site- contains configuration that is common betweenaio,stagingandproduction, e.g. enabled features, available platformssite-ha- contains configuration for the HA setup that is common betweenstagingandproduction
The environments would then be layered as follows:
base --> singlenode --> site --> aio
base --> ha --> site --> site-ha --> staging
base --> ha --> site --> site-ha --> production
with only necessary differences configured in each environment, e.g. the ingress base domain, between staging and production.
OpenStack projects
Azimuth is usually deployed on the cloud that is being targeted for workloads. It is recommended to have three OpenStack projects for a production Azimuth deployment, to contain:
- A highly-available (HA) production deployment, e.g.
azimuth-production - A HA staging deployment, e.g.
azimuth-staging - All-in-one (AIO) deployments for validating changes, e.g.
azimuth-cicd
The production and staging projects must have sufficient quota for a HA Azimuth deployment. The required quota in the CI/CD project will depend on the number of proposed changes that are open concurrently.
Users that login to Azimuth will see a list of the OpenStack projects they can access. Any OpenStack project can be used to deploy Azimuth workloads, as long as the networking requirements have been met. Azimuth creates application credentials to be able to create the user workloads using the appropriate automation.
Deployment Prerequisites
Before proceeding with an Azimuth deployment, you should ensure that you can fulfil all the prerequisites.
The prerequisites include verifying that the target cloud has the required features available, generating OpenStack application credentials and ensuring that the required DNS records and TLS certificates are available.
OpenTofu state
Azimuth deployments use OpenTofu, an open-source fork of Terraform, to manage some parts of the infrastructure.
A remote state store must be configured in order to persist the OpenTofu state across playbook executions. If GitLab is being used for the Azimuth configuration repository, it is recommended to use GitLab-managed Terraform state for this. If not, S3 is the preferred approach.
Continuous delivery
A production Azimuth deployment should use continuous delivery,
where changes to the configuration are automatically deployed to the aio and staging
environments. The ansible-playbook command should never be executed manually.
The recommended approach is to automatically deploy an independent aio environment for each
feature branch, also known as
per-branch dynamic review environments.
This allows changes to be validated before they are merged to main.
Once a change is merged to main, it will be deployed automatically to the staging environment.
Merging to main will also create a job to deploy to production, but that job will require a
manual trigger. Once the change has been validated in staging, the job to deploy to production
can be actioned.
It is also possible to run integration tests against a
deployment. It is recommended that this is configured to run automatically for at least
aio and staging environments. It should also be safe to run against production as
long as the tests are configured to target an isolated project.
A sample GitLab CI/CD configuration is provided that implements this workflow for GitLab-hosted repositories.
Monitoring
Before going into production, be sure you can access the monitoring.
Also ensure you have configured alert manager such that you will notice any alerts on staging and production.
Disaster recovery
Azimuth uses Velero to backup the data that is required to restore an Azimuth instance in the event of a catastrophic failure. This functionality is not enabled by default, as it requires credentials for an S3 bucket in which the backups will be stored.
It is recommended that disaster recovery is enabled for a production deployment.
Configuration
You are now ready to begin adding configuration to your environments. When building an environment for the first time, it is recommended to follow each documentation page in order, beginning with the Deployment method.
Tip
Remember to share as much configuration as possible between all your environments!
Created: April 9, 2024