Cross-Functional Architecture And Tools For Cloud-Based Operating Models
You can model your unique business by defining unique types of systems, service types, and foundations using the Agile Cloud Manager’s flexible Domain Specific Language DSL.
This page will go through each element of the system configuration syntax in detail, starting with a complete list of elements.
We recommend starting with our working demos and then making changes to the system configurations in our working demos instead of building from scratch. Our working demos are used for testing in our own release process.
The documentation in this section is intended to help you better understand how the working demos function, so that you can build up from the working demos.
A list of the elements of a system configuration with each element in its correct location is as follows:
<system-name>
keysDir: <value>
cloud: <value>
organization: <value>
tags: (Optional)
<any-tag-name>: <value>
<any-tag-name>: <value>
<any-tag-name>: <value>
foundation: (Optional)
instanceName: <value>
templateName: <value>
controller: <value>
…Additional fields required for some controllers
…Other optional additional fields at your discretion
preprocessor: (Optional)
locationOn: <repository-name>/<subfolder-name>/<program-or-script-name>
commandOn: <any-string-that-is-executable> $location
locationOff: <repository-name>/<subfolder-name>/<program-or-script-name>
commandOff: <any-string-that-is-executable> $location
postprocessor: (Optional)
locationOn: <repository-name>/<subfolder-name>/<program-or-script-name>
commandOn: <any-string-that-is-executable> $location
locationOff: <repository-name>/<subfolder-name>/<program-or-script-name>
commandOff: <any-string-that-is-executable> $location
mappedVariables:
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
images: (Optional)
- instanceName: <value>
templateName: <repository-name>/<subdirectory-if-present>/<template-name>
controller: <value>
mappedVariables:
<variable-name>: <value>
<variable-name>: <value>
serviceTypes:
<service-type-name>:
sharedVariables: (Optional)
mappedVariables:
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
instances:
- instanceName: <value>
templateName: <value>
controller: <value>
…Additional fields required for some controllers
…Other optional additional fields at your discretion
preprocessor: (Optional)
locationOn: <repository-name>/<subfolder-name>/<program-or-script-name>
commandOn: <any-string-that-is-executable> $location
locationOff: <repository-name>/<subfolder-name>/<program-or-script-name>
commandOff: <any-string-that-is-executable> $location
postprocessor: (Optional)
locationOn: <repository-name>/<subfolder-name>/<program-or-script-name>
commandOn: <any-string-that-is-executable> $location
locationOff: <repository-name>/<subfolder-name>/<program-or-script-name>
commandOff: <any-string-that-is-executable> $location
mappedVariables:
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
A description of each element of the system configuration is as follows:
The value of the <system-name> property must be the same as the name of the system that you specify in the acm.yaml file which will call the file that contains this system configuration. The name of the file that contains the system configuration should be <system-name>.yaml , so that the value of <system-name> should be the same in three places:
<system-name>.yaml<system-name>.yaml as shown in the example above<system-name> is calledThe keysDir property of each system gives the location of the keys.yaml file that will be used when the system is being orchestrated by the Agile Cloud Manager.
The value of the keysDir property can be any of the following options:
$Output\<name-of-folder>The value of $Default refers to a specific location that is constructed by the Agile Cloud Manager on the host computer depending on which operating system you are using.
The value of $Output\<name-of-folder> can be chosen instead if you want to specify an alternative location for keys, where <name-of-folder> is a variable whose value you define explicitly in the system configuration.
$Output\<name-of-folder> refers to /usr/acm/keys/<name-of-folder>$Output\<name-of-folder> refers to $USER_HOME\\acm\\keys\\<name-of-folder>One reason you might choose to use $Output\<name-of-folder> might be if one of your appliances has one system create new keys to be consumed by subsequent systems within the list of systems given in your acm.yaml file for the appliance.
The keysDir field in a system template will also accept literal strings that evaluate to any valid path on the host computer, but we recommend being cautious about this practice because literal strings can be brittle. Especially given the goal of making each system template widely reusable.
Remember to plan for the life cycle of anything that gets shared between systems in an appliance such as in the example above.
If the Agile Cloud Manager is running within an ephemeral agent, then any files written to keysDir will not be accessible after the agent is destroyed.
$Output\<name-of-folder> into a multiline secret in a key vault after the agent has completed a round of operations on an appliance.This refers to the name of the cloud provider that will be hosting the system that is orchestrated by the system template. This is mainly used to restrict access to certain controller logic that is cloud-specific.
The demo appliances and systems that ship with the Agile Cloud Manager give examples of how to word the cloud name for azure and aws to work with the built-in controllers.
For other clouds or on-prem or edges, you can either use cloud-agnostic building blocks, or you can create your own custom controllers to perform cloud-specific behavior, or you can contribute cloud-specific controllers to this project. If you would like to suggest naming for a given cloud that is not aws or azure, you are welcomed to send it to us so we can consider it as we develop upgrades and our own additional controllers over time.
The organization key enables you to customize the names of resources in order to avoid naming conflicts when numerous groups reuse your templates. Specifically, in the variable mapping section, you will learn that there is a command $customFunction.addOrganization.root which will append a lower case version of the value specified by the organization key to whatever string you specify as “root” in the command.
A tags block is a list of key/value pairs that can be used to populate the values of other variables in various parts of the system template.
This is intended to isolate tags that you can pass through the parameters into your basic building blocks so that resulting cloud resources can have tags associated with them.
The variable mapping section will explain how to use the $this.tags.variableName and $this.tags commands to map the values of specific tags into a basic building block.
A foundation block is optional, and contains the configuration required to orchestrate whatever shared resources you might specify to be prerequisites for the various service types in your system.
The keys in the foundation block include some that are mandatory and others that are optional.
Four keys are mandatory for every foundation. The 4 keys that are always mandatory are:
instanceNametemplateNamecontrollermappedVariablesOther keys are only mandatory when you specify certain controllers.
The structure of a foundation block is as follows:
foundation: (Optional)
instanceName: <value>
templateName: <value>
controller: <value>
…Additional fields required for some controllers
…Other optional additional fields at your discretion
preprocessor: (Optional)
…subfields specified in separate section of documentation
postprocessor: (Optional)
…subfields specified in separate section of documentation
mappedVariables: <value>
…subfields specified in separate section of documentation
images: (Optional)
- instanceName: <value>
templateName: <value>
controller: <value>
mappedVariables:
…subfields specified in separate section of documentation
A description of the fields in a foundation block is as follows:
instanceName is the unique name that you give to the foundation.
templateName is described separately below because templateName can be used in multiple places.
controller is described separately below because controller can be used in multiple places.
preprocessor and postprocessor are described separately below because they can be used in multiple places.
mappedVariables is a list of key/value pairs that we will discuss separately below because mappedVariables blocks show up in multiple locations.
images is where you organize any images that will be created and placed in your system for use by any of the compute instances that might be orchestrated by any of the instances of any of the serviceTypes in the system.
images in the foundation enable you to separate the image build process from the separate process of orchestrating services.images section is a list of images which each contain at least 4 mandatory fields and some optional fields.The controller-specific mandatory key/value pairs will be discussed separately below because they also show up in multiple locations.
The optional key/value pairs are intended to be used as inputs for the variable mapping that is described in a different section. For example, $this.foundation.variableName and $this.foundation in allowed locations can refer to an optional key/value pair that you might add to the foundation.
A serviceTypes block is a list of named serviceType definitions which together compose the group of types of services included in the system.
An example of the format of a serviceTypes section is:
serviceTypes:
<service-type-name>:
sharedVariables: (Optional)
mappedVariables:
…subfields specified in separate section of documentation
instances:
- instanceName: <value>
templateName: <value>
controller: <value>
…Additional fields required for some controllers
…Other optional additional fields at your discretion
preprocessor: (Optional)
…subfields specified in separate section of documentation
postprocessor: (Optional)
…subfields specified in separate section of documentation
mappedVariables:
…subfields specified in separate section of documentation
A description of the high-level fields in a serviceTypes block is:
<service-type-name> is where you put the unique name that you choose for each type of service.
sharedVariables is an optional block within which you can place a mappedVariables block for variable mappings that get reused by each instance of the type of service in one place for brevity. The Agile Cloud Manager’s algorithms will merge the contents of this shared mappedVariables block with each of the instance mappedVariables blocks at runtime so that you can avoid redundant variable mappings.
instances is a list of definitions of each of the instances of the specific type of service. The contents of the instances section will be described separately below.
Each instance of a given serviceType is listed individually with 4 mandatory keys and any number of optional keys. Some of the optional keys will be required if and only if you use a specific controller.
The 4 required keys are instanceName, templateName, controller, and mappedVariables.
instanceName is the unique name that you give to the instance of the serviceType.
templateName is described separately below because templateName can be used in multiple places.
controller is described separately below because controller can be used in multiple places.
mappedVariables is a list of key/value pairs that we will discuss separately below because mappedVariables blocks show up in multiple locations.
The controller-specific mandatory variables will be discussed separately below because they also show up in multiple locations.
The following key names can be used in multiple locations as specified above. Each of these following key names is being described here separately because each of the following key names is handled the same way regardless of which of the legal locations it is located in.
templateName is the relative path to the basic building block template that will be used by the controller to create cloud resources.
Since the Agile Cloud Manager clones repositories into sibling directories of the directory in which the acm.yaml appliance definition is located, the templateName will be the relative path from the parent directory to the template, and the first folder name will be the name of the repository.
You should examine the working demo examples that ship with the Agile Cloud Manager in order to understand how this works, and to start with something that works. But one example of a templateName value might be:
aws-building-blocks/cf/ec2withcustomimg.json
controller specifies which controller will be used to orchestrate the foundation.
This can either be one of the built-in controllers or can be a custom controller that you might develop.
For built-in controllers, you use the string name of the controller, such as:
For custom controllers, you have two options, depending on whether you are using an API custom controller or a module custom controller.
$customController.acm-custom-controller/controller/customController.pyNote: API custom controllers are new in version 1.1 of Agile Cloud Manager, so they will only work in version 1.1 and higher.
The preprocessor block is where you specify your own custom programs to run before a foundation or before an instance of a service type is created, updated, or deleted.
The postprocessor block is where you specify your own custom programs to run after a foundation or after an instance of a service type is created, updated, or deleted.
preprocessor and postprocessor can be used for many purposes.
You can use preprocessor and postprocessor for some of the gating that gets done between stages of legacy pipelines, a place where you can automate checks for conditions that your governance requires to be met at specific points in your deployment processes.
You can also use preprocessor and postprocessor to signal other systems to take specific actions in response to specific changes being made when a foundation or a serviceInstance is created, updated, or deleted.
The syntax for preprocessor and postprocessor can be as follows:
preprocessor:
locationOn: azure-building-blocks/scripts/hello1.py
commandOn: python $location
locationOff: azure-building-blocks/scripts/hello2.py
commandOff: python $location
postprocessor:
locationOn: azure-building-blocks/scripts/hello3.py
commandOn: python $location
locationOff: azure-building-blocks/scripts/hello4.py
commandOff: python $location
In the example above, locationOn gives the location of the script to run when the on command is run, for example acm appliance on, or acm foundation on, or acm serviceType on, or acm serviceInstance on.
And locationOff gives the location of the script to run when the off command is run, for example acm appliance off, or acm foundation off, or acm serviceType off, or acm serviceInstance off.
The syntax for locationOn and for locationOff is the same as for the templateName described elsewhere, in the sense that it is the relative path to the program from the perspective of the parent folder of the folder that contains the acm.yaml appliance definition.
If you look at this another way, remember that the acm setup on command clones all related repositories into sibling directories to the directory which contains acm.yaml. Therefore, the syntax for locationOn and locationOff is <repositoryName>/<subfolderIfPresent>/<programName> .
The syntax for commandOn and commandOff is an example of string interpolation. Specifically, in the example python $location given above, $location is the interpolated value of either locationOn or of locationOff, so that for the postprocessor commandOff in the example above, python $location would evaluate to python azure-building-blocks/scripts/hello4.py and the Agile Cloud Manager would run the command from the parent folder so that the correct azure-building-blocks/scripts/hello4.py program would be run.
commandOn and commandOff can be written as any string you want, as long as the result is an executable command. For example, you can specify any other program if you do not want to use python.
A mappedVariables block is present in every foundation, is present in every instance of each serviceType, and can also be in an optional sharedVariables section for each serviceType.
The mappedVariables sections are the primary location where the variableMapping syntax is used because mappings specified in the mappedVariables section get fed into a “command builder” component of the Agile Cloud Manager’s source code which translates the mappings into commands that can be understood by each of the third-party tools that instantiate basic building blocks.
Each mappedVariables block takes the following form:
mappedVariables:
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
<variable-name>: <value>
The values for each of the keys in the mappedVariables list are specified using the variable mapping syntax described in a separate section entitled “Variable Mapping Syntax for the Agile Cloud Manager Domain Specific Language”.
In addition to the variables that are required for any foundation or for any instance of any serviceType, there are also some variables that are required only when you use certain controllers.
A discussion of some of the controllers with specific syntax requirements is as follows:
The arm controller requires 4 additional fields in addition to the fields that are required for other controllers.
The additional required fields are described as follows:
deploymentName is the name of the Azure deployment object to be created when the ARM template is run. This can be a string.
emptyTemplateName is the path to an empty ARM template that will be used when off commands are run.
templateName is used for emptyTemplateName.resourceGroupName specifies the resource group in which to create the resources that are defined in the ARM template. This can be a string.
resourceGroupRegion specifies the region in which to create the resource group. This can be a string.
An example of correct syntax for using the arm controller is as follows:
instanceName: <value>
deploymentName: <value>
templateName: <value>
emptyTemplateName: <value>
controller: arm
resourceGroupName: <value>
resourceGroupRegion: <value>
preprocessor: (Optional)
…subfields specified in separate section of documentation
postprocessor: (Optional)
…subfields specified in separate section of documentation
mappedVariables:
…subfields specified in separate section of documentation
Two items are specifically dedicated to the terraform controller, including backendVariables and forceDelete.
The terraform controller requires a backendVariables block in addition to the basic fields that are required for other types of instances.
The backendVariables block is a list of 4 required key/value pairs. The four required field names are:
storage_account_namecontainer_namekeyaccess_keyThe values in a backendVariables block can either be strings, or can be sourced from keys.yaml. If you source a value from keys.yaml, the syntax depends on whether the field name is the same in the backendVariables block as it is in keys.yaml. If the field name is the same, you simply use $keys. But if the field name is different in the backendVariables block than in keys.yaml, then you use $keys.variableName, where variableName refers to the name of the keys.yaml field from which you want to source the value.
An example of proper syntax for using the terraform controller including a backendVariables block is as follows:
instanceName: <value>
templateName: <value>
controller: terraform
mappedVariables:
…subfields specified in separate section of documentation
backendVariables:
storage_account_name: $keys.variableName
container_name: $keys
key: <any-string>
access_key: $keys.variableName
The “forceDelete: True” key is experimental and is provided with the intention of soliciting feedback from users about how it might most effectively be defined in future versions.
As such, “forceDelete: True” only works in the specific situation for which it was designed and tested as described below.
“forceDelete: True” was created to protect service types named “tfBackend” from being destroyed by automation unless you explicitly tell the system configuration to destroy the tfBackend by setting “forceDelete: True”.
Note that “tfBackend” is case sensitive.
The working demos that ship with the Agile Cloud Manager should be used as starting points if you want to use “forceDelete: True”.
The working example of “forceDelete: True” is:
This demo using “forceDelete: True” works out-of-the-box. After you get this demo working, then you can examine whether or not you need different behavior.
Do not use “forceDelete: True” for any other use cases besides the working example that is given.
There are many cases where “forceDelete: True” will not work by design, and where you might even end up with unexpectedly orphaned service types that are not deleted when you want them deleted.
You are welcomed to submit a feature request if you want additional specific functionality. You are also welcomed to participate in our weekly meetings of developers who want to contribute to this project.
Using the cloudformation controller requires adding a stackName field for each instance, and an imageName field only for image instances.
stackName refers to the stack that will be created to organize the deployment using the cloudformation template that you specify in the templateName field.
imageName defines the root string of the name of the image that will be created. The automation may add a datetime suffix to the value specified for imageName to support many iterative versions of resulting images, for example if you run acm foundation on <flags> many times for a given system. Start with one of our working demos to see how imageName works with the cloudformation controller.
An example of correct syntax for using the cloudformation controller is:
instanceName: <value>
stackName: <value>
templateName: <repository-name/<subdirectory-if-present>/ec2withcustomimg.json
controller: cloudformation
preprocessor:
…subfields specified in separate section of documentation
postprocessor:
…subfields specified in separate section of documentation
mappedVariables:
…subfields specified in separate section of documentation
Of course, your template name might not be ec2withcustomimg.json, but the point in the example is to illustrate the format.
And of course, the imageName field would need to be added if and only if the block defines an image. See the working demo for a working example.
Using a custom controller involves first creating a custom controller of your own and then making changes to the standard syntax in the system configuration file. The changes depend on whether you are using an API custom controller or a module custom controller.
For an API custom controller, you specify the controller key with a value that follows the syntax $customControllerAPI.<port-number-on-localhost>/<rest-of-path-to-api-endpoint>/ . For example, in our demo, we use the syntax: $customControllerAPI.8675/acm/controller/custom/
For a module custom controller, you must specify two changes as follows:
controller must be specified using the syntax $customController.<repository-name>/<subdirectory-if-present>/<name-of-controller-script-or-program> to indicate that it is a custom controller.
A controllerCommand field must be added with the syntax <command> $location. You can use any value you want for <command> as long as the result of <command> $location is executable. For example, you can program in a different language if you do not want to use python.
An example of proper syntax for calling a module custom controller from a system template is as follows:
instanceName: <value>
templateName: <repository-name>/<subdirectory-if-present>/<name-of template-file>.json
controller: $customController.<repository-name>/<subdirectory-if-present>/<name-of-controller-script-or-program>
controllerCommand: python $location
preprocessor:
…subfields specified in separate section of documentation
postprocessor:
…subfields specified in separate section of documentation
mappedVariables:
…subfields specified in separate section of documentation
Start with our working examples of custom controllers and then make changes if you decide to build a custom controller. The working demos are intended to save you a lot of time getting things working right away.