Cross-Functional Architecture And Tools For Cloud-Based Operating Models
The simple, yet flexible, structure of the Agile Cloud Manager’s Domain Specific Language DSL gives you the freedom to define your own customized and reusable types for each building block of your platforms.
This article will explain some of the elements which might be considered in the Low-Level Designs that you might create for implementations of the Agile Cloud Manager.
Low Level Design LLD refers to the more granular level of how code and configuration will be structured in a solution to meet the requirements that have been defined in higher-level architectural documents.
LLD documents are usually prepared either by a senior engineer or by a junior architect.
For the Agile Cloud Manager, Low Level Design involves designing the structure of all the different types of code and configuration files given in the diagram entitled “Code and Configuration Orchestrated by the Agile Cloud Manager”.
The diagram illustrates the relationships between each of the 7 types of code and configuration that must be considered in your low-level designs, and which might be sourced from 4 different types of locations also shown in the diagram.
The remaining two subsections of this article will explain the elements of the diagram in more detail.
The 4 types of source locations shown from left to right in the diagram include:
Three items come from any pipeline tool of your choice. These items include:
Your pipeline tool will place keys.yaml and config.yaml into each of the agents orchestrated by any given workflow. Your pipeline tool can source the contents of keys.yaml and of config.yaml from any source you specify.
The appliance/domain repository contains only two files.
This is where your organization will store all the reusable templates for system definitions to be shared across the entire enterprise.
All these system definitions will be written in the Agile Cloud Manager’s simple and flexible Domain Specific Language DSL.
In practice, this repository might be a reduced subset of templates that might be the result of a build process that might publish only subsets of the enterprise library to each team for security reasons.
These are the repositories where the entire library of basic building blocks of third-party templates might be stored.
The Agile Cloud Manager ships with a separate repository for each cloud provider, choosing to bundle all building blocks for a given cloud provider into a separate repository.
Your organization might also choose to publish only subsets of your enterprise library of basic building blocks to each team for security reasons.
The 7 types of code and configuration in the order given in the diagram above are as follows:
Number 1 in the diagram is acm.yaml, the central file which represents an appliance which is composed of one or more custom-defined systems.
An acm.yaml could be as simple as a list of a few lines, each of which contain:
An example of the format for acm.yaml is as follows:
firstSystemName: acm-system-config-repo-name/firstSystemName.yaml secondSystemName: acm-system-config-repo-name/secondSystemName.yaml thirdSystemName: acm-system-config-repo-name/thirdSystemName.yaml fourthSystemName: acm-system-config-repo-name/fourthSystemName.yaml
A real acm.yaml would of course use actual names for each type of system and for each file, and might look like the example in the following diagram.
The diagram points to and explains each of 4 components of an example acm.yaml file.
acm.yaml is a very powerful file, because only a few lines of code orchestrate an entire appliance as part of an enterprise pipeline.
The structure of the git repository for the above example would look like:
And the structure of the ` acm-system-config-templates` repository containing all the system configuration templates would therefore look like:
storage.yaml database.yaml microservices.yaml ui.yaml many other system config yaml files. …
Notice that the system configuration templates are in a separate repository so that system configurations can be re-used and shared throughout the enterprise.
Your enterprise might define subscription rules so that one central repository of all system templates might publish only narrow subsets of its template library to each consuming group for security reasons.
If you look again at the top of this article to the diagram entitled “Code and Configuration Orchestrated By The Agile Cloud Manager”, number 2 in that diagram is system configuration.
You can define a custom system configuration with a custom name to model each system in your organization in a way that is relevant to your unique business.
An example of a system configuration file might look like:
<system-name> keysDir: <value> forceDelete: <value> (Optional) cloud: <value> organization: tags: (Optional) <any-tag-name>: <value> <any-tag-name>: <value> <any-tag-name>: <value> foundation: (Optional) instanceName: <value> templateName: <value> controller: <value> mappedVariables: <value> <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> backendVariables: (Optional) <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> serviceTypes: <service-type-name>: sharedVariables: mappedVariables: <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> instances: - instanceName: <value> templateName: <value> type: <value> controller: <value> <other-variables-with-any-values>:<value> mappedVariables: <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> backendVariables: (Optional) <variable-name>: <value> <variable-name>: <value> <variable-name>: <value> <variable-name>: <value>
A simple variable-mapping syntax enables you to tell the Agile Cloud Manager where to find the values for each of the variables that you can define in the
mappedVariables section and in the
backendVariables sections of system configuration blocks like the one shown above.
Altogether, variable mapping enables you to create concise, reusable summaries of each of your organization’s unique systems so that it becomes easier to align your IT investments with your business objectives.
A more detailed description of the syntax for system configuration is given in the article entitled “System Configuration Specification for the Agile Cloud Manager Domain Specific Language”.
A detailed description of the simple variable-mapping syntax is in our article entitled “Variable Mapping Syntax for the Agile Cloud Manager Domain Specific Language”.
Number 3 in the chart above entitled “Code and Configuration Orchestrated By The Agile Cloud Manager” refers to your pre-existing third party templates for:
The Agile Cloud Manager ships with demonstration code to orchestrate Terraform, Packer, ARM templates, CloudFormation templates, and custom templates using custom controllers that can be extended to orchestrate any other type of template.
It is also very easy to integrate the Agile Cloud Manager with configuration-management tools, as described in our separate article entitled “Integrating the Agile Cloud Manager with Ansible, Chef, or Puppet.”
The demonstration code building blocks that we provide separate your third-party templates into categories that align with the Agile Cloud Manager’s Domain Specific Language DSL, which you can read about in the section about in the sections of this site that discuss system configuration. In short, the three ways we categorize your third-party basic building block templates are:
You probably already have many redundant versions of templates for third party infrastructure-as-code, configuration-as-code, and SaaS definitions.
Low Level Design therefore includes examining all your existing templates and cloud implementations with the objective of streamlining them into a smaller subset of parameterized, reusable templates that can enable you to derive the greatest financial benefits from the Agile Cloud Manager.
Number 4 in the diagram above entitled “Code and Configuration Orchestrated By The Agile Cloud Manager” is a file called config.yaml.
The great thing about config.yaml is that it allows you to make your system templates completely reusable. config.yaml is a flat list of key/value pairs that you can use as a source for values that the Agile Cloud Manager can map into specific fields within system templates at runtime.
For example, you can use config.yaml to specify values for variables that underlying cloud providers use as unique names of cloud resources such as storage names and resource group names.
By enabling the passing of unique names into the same system templates, config.yaml allows each system template to become widely reusable throughout your organization.
The section of this web site about variable mapping rules will explain how and where within each system template you can source values from config.yaml.
Number 5 in the diagram above entitled “Code and Configuration Orchestrated By The Agile Cloud Manager” is a file called keys.yaml, which contains all the secrets that also make it easier to re-use your system definitions.
keys.yaml follows a very simple syntax that is nothing more than a flat list of key/value pairs with strings as values, so that each key/value pair is on its own line in the simplest flat yaml.
A secure build process must be defined for your keys.yaml.
Your pipeline code will need to pull the secrets in from a secure location such as a key vault, and then place the secrets into a keys.yaml file located in the path within each agent specified by your system configuration. Your automation can also lock down the path containing keys.yaml and can then later delete the file after it is no longer needed.
Number 6 in the diagram above entitled “Code and Configuration Orchestrated By The Agile Cloud Manager” refers to the CLI commands offered by the Agile Cloud Manager and to the Pipeline workflows which will employ the Agile Cloud Manager’s CLI commands in your organization.
The Agile Cloud Manager has a very simple set of CLI commands which are designed to keep the amount of code in calling scripts very brief so that it is easy to change back and forth between any pipeline tool.
In fact, there are only 12 CLI commands as follows:
acm setup on sourceRepo=<url-of-appliance-repo> acm appliance on acm appliance off acm foundation on systemName=<name-of-system> acm services on systemName=<name-of-system> acm services off systemName=<name-of-system> acm serviceType on systemName=<name-of-system> serviceType=<name-of-type> acm serviceType off systemName=<name-of-system> serviceType=<name-of-type> acm serviceInstance on systemName=<name> serviceType=<name> serviceInstance=<name> acm serviceInstance off systemName=<name> serviceType=<name> serviceInstance=<name> acm foundation off systemName=<name-of-system> acm setup off
The CLI commands are intended to be used in specific sequences in workflows because they have dependencies upon each other.
Also, the flags given in each CLI command shown above are required and are not optional.
A more detailed description of the syntax and usage rules for CLI commands is given in our article entitled “CLI Commands and Workflows for the Agile Cloud Manager Domain Specific Language”.
Number 7 in the diagram entitled “Code and Configuration Orchestrated By The Agile Cloud Manager” refers to a file named setupConfig.yaml, which defines all the steps that the Agile Cloud Manager will follow when configuring itself to run commands on a computer.
A working example of setupConfig.yaml is distributed with each of our appliance working examples, so it is better for you to start from a working example of setupConfig.yaml and to modify the working example as needed instead of writing a new version of the file yourself.
The structure of the setupConfig.yaml file is:
dependencies: - name: azure-cli version: "2.33" download-link-windows: download-link-linux: modules: - name: azure-cli-ext-azure-devops version: "0.23" - name: git version: "2" download-link-windows: https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.27.0.tar.gz download-link-linux: checksums: - url: https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.27.0.tar.sign - url: https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.27.0.tar.xz - name: terraform version: "0.14" download-link-windows: https://releases.hashicorp.com/terraform/0.14.3/terraform_0.14.3_windows_amd64.zip download-link-linux: https://releases.hashicorp.com/terraform/0.14.3/terraform_0.14.3_linux_amd64.zip checksums: - url: https://releases.hashicorp.com/terraform/0.14.3/terraform_0.14.3_SHA256SUMS - url: https://releases.hashicorp.com/terraform/0.14.3/terraform_0.14.3_SHA256SUMS.348FFC4C.sig - url: https://releases.hashicorp.com/terraform/0.14.3/terraform_0.14.3_SHA256SUMS.72D7468F.sig - url: https://releases.hashicorp.com/terraform/0.14.3/terraform_0.14.3_SHA256SUMS.sig - name: packer version: "1.7" download-link-windows: https://releases.hashicorp.com/packer/1.7.3/packer_1.7.3_windows_amd64.zip download-link-linux: https://releases.hashicorp.com/packer/1.7.3/packer_1.7.3_linux_amd64.zip checksums: - url: https://releases.hashicorp.com/packer/1.7.3/packer_1.7.3_SHA256SUMS - url: https://releases.hashicorp.com/packer/1.7.3/packer_1.7.3_SHA256SUMS.sig source: - instanceName: azure-building-blocks repo: https://github.com/AgileCloudInstitute/azure-building-blocks.git branch: main - instanceName: aws-building-blocks repo: https://github.com/AgileCloudInstitute/aws-building-blocks.git branch: main - instanceName: acm-system-templates repo: https://github.com/AgileCloudInstitute/acm-system-templates.git branch: main
As you can see, setupConfig.yaml has a
dependencies section and a
dependencies section specifies some of the dependencies that the
acm setup on sourceRepo=<url-of-appliance-repo> command can check for during setup, including the required versions for each of the covered dependencies. Currently, only the dependencies listed in our various working demo setupConfig.yaml examples are supported for this checking feature.
To use setupConfig.yaml, begin with one of our working examples and then experiment with changing version numbers and download URLs, etc. in the working examples.
source block defines the repositories that the Agile Cloud Manager will download during the setup process for use when running the Agile Cloud Manager’s CLI commands. As you can see from the example above, two types of repositories are listed in the
source block, including:
These two types of repositories support the ability of your organization to curate organization-wide standards for reusable templates starting with basic building blocks and working all the way up to an enterprise value stream pipeline composed of appliances defined by a series of acm.yaml files and the CLI commands exposed by the Agile Cloud Manager.
To understand how these various elements of the Low-Level Design fit into higher level enterprise architecture, you can read our other article entitled “Architecting the Agile Enterprise with the Agile Cloud Manager”
Finally, note that
acm setup on <flags> expects that certain dependencies will already be installed on your machine, but that
acm setup on <flags> will also install only certain other dependencies, depending on how you have configured setupConfig.yaml. For example:
acm setup on <flags>command will only check to confirm that the correct versions of these dependencies have been installed:
acm setup on <flags>IF you have defined these following tools properly in setupConfig.yaml: