Low-Level Design with the Agile Cloud Manager
The simple, yet flexible, structure of the Agile Cloud Manager’s Domain Specific Language DSL gives you the freedom to define your own customized and reusable types for each building block of your platforms.
This article will explain some of the elements which might be considered in the Low-Level Designs that you might create for implementations of the Agile Cloud Manager.
Low-Level Design Elements
Low Level Design LLD refers to the more granular level of how code and configuration will be structured in a solution to meet the requirements that have been defined in higher-level architectural documents.
LLD documents are usually prepared either by a senior engineer or by a junior architect.
For the Agile Cloud Manager, Low Level Design involves designing the structure of all the different types of code and configuration files given in the diagram entitled “Code and Configuration Orchestrated by the Agile Cloud Manager”.
The diagram illustrates the relationships between each of the 7 types of code and configuration that must be considered in your low-level designs, and which might be sourced from 4 different types of locations also shown in the diagram.
The remaining two subsections of this article will explain the elements of the diagram in more detail.
Four Source Locations
The 4 types of source locations shown from left to right in the diagram include:
ANY Pipeline Tool
Three items come from any pipeline tool of your choice. These items include:
- Workflows that call the CLI commands of the Agile Cloud Manager to operate on elements of the appliance.
- keys.yaml, which is a secrets file that can be sourced as a multi-line secret from a key vault and then written into an agent at runtime.
- config.yaml, which is list of key/value pairs that enable each of your system templates to become widely reusable.
Your pipeline tool will place keys.yaml and config.yaml into each of the agents orchestrated by any given workflow. Your pipeline tool can source the contents of keys.yaml and of config.yaml from any source you specify.
Repository containing the Appliance/Domain Managed by the Agile Cloud Manager
The appliance/domain repository contains only two files.
- acm.yaml is a very simple file that summarizes the appliance/domain in perhaps only a few lines.
- setupConfig.yaml defines where the Agile Cloud Manager must look to find both system configuration templates and basic building blocks templates. setupConfig.yaml also defines versions of some of the dependencies to be checked during the setup process. Note that you must do some separate provisioning in addition to what setupConfig.yaml configures. The section of this web site that elaborates on setupConfig.yaml will explain what other provisioning you must do.
Repository containing the System Configurations Library using the Agile Cloud Manager DSL
This is where your organization will store all the reusable templates for system definitions to be shared across the entire enterprise.
All these system definitions will be written in the Agile Cloud Manager’s simple and flexible Domain Specific Language DSL.
In practice, this repository might be a reduced subset of templates that might be the result of a build process that might publish only subsets of the enterprise library to each team for security reasons.
Repositories containing Third Party Building Block Templates
These are the repositories where the entire library of basic building blocks of third-party templates might be stored.
The Agile Cloud Manager ships with a separate repository for each cloud provider, choosing to bundle all building blocks for a given cloud provider into a separate repository.
Your organization might also choose to publish only subsets of your enterprise library of basic building blocks to each team for security reasons.
Types Of Code And Configuration
The 7 types of code and configuration in the order given in the diagram above are as follows:
acm.yaml
Number 1 in the diagram is acm.yaml, the central file which represents an appliance which is composed of one or more custom-defined systems.
An acm.yaml could be as simple as a list of a few lines, each of which contain:
- The name of a system.
- The name of the file where the system config for that system can be located.
- The name of the repository in which the template is located.
An example of the format for acm.yaml is as follows:
firstSystemName: acm-system-config-repo-name/firstSystemName.yaml
secondSystemName: acm-system-config-repo-name/secondSystemName.yaml
thirdSystemName: acm-system-config-repo-name/thirdSystemName.yaml
fourthSystemName: acm-system-config-repo-name/fourthSystemName.yaml
A real acm.yaml would of course use actual names for each type of system and for each file, and might look like the example in the following diagram.
The diagram points to and explains each of 4 components of an example acm.yaml file.
acm.yaml is a very powerful file, because only a few lines of code orchestrate an entire appliance as part of an enterprise pipeline.
The structure of the git repository for the above example would look like:
acm.yaml
setupConfig.yaml
And the structure of the ` acm-system-config-templates` repository containing all the system configuration templates would therefore look like:
storage.yaml
database.yaml
microservices.yaml
ui.yaml
many other system config yaml files. …
Notice that the system configuration templates are in a separate repository so that system configurations can be re-used and shared throughout the enterprise.
Your enterprise might define subscription rules so that one central repository of all system templates might publish only narrow subsets of its template library to each consuming group for security reasons.
System Configuration
If you look again at the top of this article to the diagram entitled “Code and Configuration Orchestrated By The Agile Cloud Manager”, number 2 in that diagram is system configuration.
You can define a custom system configuration with a custom name to model each system in your organization in a way that is relevant to your unique business.
An example of a system configuration file might look like:
<system-name>
keysDir: <value>
forceDelete: <value> (Optional)
cloud: <value>
organization:
tags: (Optional)
<any-tag-name>: <value>
<any-tag-name>: <value>
<any-tag-name>: <value>
foundation: (Optional)
instanceName: <value>
templateName: <value>
controller: <value>
mappedVariables: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
backendVariables: (Optional)
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
serviceTypes:
<service-type-name>:
sharedVariables:
mappedVariables:
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
instances:
- instanceName: <value>
templateName: <value>
type: <value>
controller: <value>
<other-variables-with-any-values>:<value>
mappedVariables:
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
backendVariables: (Optional)
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
A simple variable-mapping syntax enables you to tell the Agile Cloud Manager where to find the values for each of the variables that you can define in the mappedVariables section and in the backendVariables sections of system configuration blocks like the one shown above.
Altogether, variable mapping enables you to create concise, reusable summaries of each of your organization’s unique systems so that it becomes easier to align your IT investments with your business objectives.
A more detailed description of the syntax for system configuration is given in the article entitled “System Configuration Specification for the Agile Cloud Manager Domain Specific Language”.
A detailed description of the simple variable-mapping syntax is in our article entitled “Variable Mapping Syntax for the Agile Cloud Manager Domain Specific Language”.
Templates of Building Blocks
Number 3 in the chart above entitled “Code and Configuration Orchestrated By The Agile Cloud Manager” refers to your pre-existing third party templates for:
- infrastructure-as-code
- configuration-as-code
- SaaS definitions
The Agile Cloud Manager ships with demonstration code to orchestrate Terraform, Packer, ARM templates, CloudFormation templates, and custom templates using custom controllers that can be extended to orchestrate any other type of template.
It is also very easy to integrate the Agile Cloud Manager with configuration-management tools, as described in our separate article entitled “Integrating the Agile Cloud Manager with Ansible, Chef, or Puppet.”
The demonstration code building blocks that we provide separate your third-party templates into categories that align with the Agile Cloud Manager’s Domain Specific Language DSL, which you can read about in the section about in the sections of this site that discuss system configuration. In short, the three ways we categorize your third-party basic building block templates are:
- Foundation templates
- Image templates
- Service type templates
You probably already have many redundant versions of templates for third party infrastructure-as-code, configuration-as-code, and SaaS definitions.
Low Level Design therefore includes examining all your existing templates and cloud implementations with the objective of streamlining them into a smaller subset of parameterized, reusable templates that can enable you to derive the greatest financial benefits from the Agile Cloud Manager.
config.yaml
Number 4 in the diagram above entitled “Code and Configuration Orchestrated By The Agile Cloud Manager” is a file called config.yaml.
The great thing about config.yaml is that it allows you to make your system templates completely reusable. config.yaml is a flat list of key/value pairs that you can use as a source for values that the Agile Cloud Manager can map into specific fields within system templates at runtime.
For example, you can use config.yaml to specify values for variables that underlying cloud providers use as unique names of cloud resources such as storage names and resource group names.
By enabling the passing of unique names into the same system templates, config.yaml allows each system template to become widely reusable throughout your organization.
The section of this web site about variable mapping rules will explain how and where within each system template you can source values from config.yaml.
keys.yaml
Number 5 in the diagram above entitled “Code and Configuration Orchestrated By The Agile Cloud Manager” is a file called keys.yaml, which contains all the secrets that also make it easier to re-use your system definitions.
keys.yaml follows a very simple syntax that is nothing more than a flat list of key/value pairs with strings as values, so that each key/value pair is on its own line in the simplest flat yaml.
A secure build process must be defined for your keys.yaml.
Your pipeline code will need to pull the secrets in from a secure location such as a key vault, and then place the secrets into a keys.yaml file located in the path within each agent specified by your system configuration. Your automation can also lock down the path containing keys.yaml and can then later delete the file after it is no longer needed.
CLI Commands and Pipeline Workflows
Number 6 in the diagram above entitled “Code and Configuration Orchestrated By The Agile Cloud Manager” refers to the CLI commands offered by the Agile Cloud Manager and to the Pipeline workflows which will employ the Agile Cloud Manager’s CLI commands in your organization.
The Agile Cloud Manager has a very simple set of CLI commands which are designed to keep the amount of code in calling scripts very brief so that it is easy to change back and forth between any pipeline tool.
In fact, there are only 12 CLI commands as follows:
acm setup on sourceRepo=<url-of-appliance-repo>
acm appliance on
acm appliance off
acm foundation on systemName=<name-of-system>
acm services on systemName=<name-of-system>
acm services off systemName=<name-of-system>
acm serviceType on systemName=<name-of-system> serviceType=<name-of-type>
acm serviceType off systemName=<name-of-system> serviceType=<name-of-type>
acm serviceInstance on systemName=<name> serviceType=<name> serviceInstance=<name>
acm serviceInstance off systemName=<name> serviceType=<name> serviceInstance=<name>
acm foundation off systemName=<name-of-system>
acm setup off
The CLI commands are intended to be used in specific sequences in workflows because they have dependencies upon each other.
Also, the flags given in each CLI command shown above are required and are not optional.
A more detailed description of the syntax and usage rules for CLI commands is given in our article entitled “CLI Commands and Workflows for the Agile Cloud Manager Domain Specific Language”.
setupConfig.yaml
Number 7 in the diagram entitled “Code and Configuration Orchestrated By The Agile Cloud Manager” refers to a file named setupConfig.yaml, which defines all the steps that the Agile Cloud Manager will follow when configuring itself to run commands on a computer.
A working example of setupConfig.yaml is distributed with each of our appliance working examples, so it is better for you to start from a working example of setupConfig.yaml and to modify the working example as needed instead of writing a new version of the file yourself.
The structure of the setupConfig.yaml file is:
dependencies:
- name: azure-cli
version: "2.33"
download-link-windows:
download-link-linux:
modules:
- name: azure-cli-ext-azure-devops
version: "0.23"
- name: git
version: "2"
download-link-windows: https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.27.0.tar.gz
download-link-linux:
checksums:
- url: https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.27.0.tar.sign
- url: https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.27.0.tar.xz
- name: terraform
version: "0.14"
download-link-windows: https://releases.hashicorp.com/terraform/0.14.3/terraform_0.14.3_windows_amd64.zip
download-link-linux: https://releases.hashicorp.com/terraform/0.14.3/terraform_0.14.3_linux_amd64.zip
checksums:
- url: https://releases.hashicorp.com/terraform/0.14.3/terraform_0.14.3_SHA256SUMS
- url: https://releases.hashicorp.com/terraform/0.14.3/terraform_0.14.3_SHA256SUMS.348FFC4C.sig
- url: https://releases.hashicorp.com/terraform/0.14.3/terraform_0.14.3_SHA256SUMS.72D7468F.sig
- url: https://releases.hashicorp.com/terraform/0.14.3/terraform_0.14.3_SHA256SUMS.sig
- name: packer
version: "1.7"
download-link-windows: https://releases.hashicorp.com/packer/1.7.3/packer_1.7.3_windows_amd64.zip
download-link-linux: https://releases.hashicorp.com/packer/1.7.3/packer_1.7.3_linux_amd64.zip
checksums:
- url: https://releases.hashicorp.com/packer/1.7.3/packer_1.7.3_SHA256SUMS
- url: https://releases.hashicorp.com/packer/1.7.3/packer_1.7.3_SHA256SUMS.sig
source:
- instanceName: azure-building-blocks
repo: https://github.com/AgileCloudInstitute/azure-building-blocks.git
branch: main
- instanceName: aws-building-blocks
repo: https://github.com/AgileCloudInstitute/aws-building-blocks.git
branch: main
- instanceName: acm-system-templates
repo: https://github.com/AgileCloudInstitute/acm-system-templates.git
branch: main
As you can see, setupConfig.yaml has a dependencies section and a source section.
The dependencies section specifies some of the dependencies that the acm setup on sourceRepo=<url-of-appliance-repo> command can check for during setup, including the required versions for each of the covered dependencies. Currently, only the dependencies listed in our various working demo setupConfig.yaml examples are supported for this checking feature.
To use setupConfig.yaml, begin with one of our working examples and then experiment with changing version numbers and download URLs, etc. in the working examples.
The source block defines the repositories that the Agile Cloud Manager will download during the setup process for use when running the Agile Cloud Manager’s CLI commands. As you can see from the example above, two types of repositories are listed in the source block, including:
- Basic building block repositories that contain third-party templates for infrastructure-as-code, for configuration-as-code, and SaaS definitions for specific cloud providers.
- A system configuration template repository that contains all the system definitions for an organization that have been defined using the Agile Cloud Manager’s Domain Specific Language DSL.
These two types of repositories support the ability of your organization to curate organization-wide standards for reusable templates starting with basic building blocks and working all the way up to an enterprise value stream pipeline composed of appliances defined by a series of acm.yaml files and the CLI commands exposed by the Agile Cloud Manager.
To understand how these various elements of the Low-Level Design fit into higher level enterprise architecture, you can read our other article entitled “Architecting the Agile Enterprise with the Agile Cloud Manager”
Finally, note that acm setup on <flags> expects that certain dependencies will already be installed on your machine, but that acm setup on <flags> will also install only certain other dependencies, depending on how you have configured setupConfig.yaml. For example:
- Your own separate provisioning scripts must already have installed the following tools if you define them in setupConfig.yaml, and the
acm setup on <flags>command will only check to confirm that the correct versions of these dependencies have been installed:- git
- aws cli
- az cli
- The following tools will be installed by the Agile Cloud Manager when you run
acm setup on <flags>IF you have defined these following tools properly in setupConfig.yaml:- Terraform
- Packer