Getting Started for Engineers (Pipeline Version)
After after you have created a DevBox by following the instructions at this other link, you will be ready to set up a CICD pipeline using the Agile Cloud Manager by following the instructions in this document.
Any pipeline tool can be used for CICD with the Agile Cloud Manager. The reason we are using GitHub in this example is that GitHub is widely used. When you have the GitHub version up and running, you will have a working example that can make it easier for you to set up any other pipeline tool.
The following steps describe how to get a CICD pipeline for the example applications working in GitHub:
- Assemble all the values required by keys.yaml and config.yaml in order for the demos to function properly.
- Open cloud provider dashboards to monitor demos.
- Fork the GitHub demos repository.
- Put the config.yaml and keys.yaml values into two new multiline secrets in GitHub.
- Run all 4 of the demo workflows in the “Actions” tab of your repository as described below.
- Monitor Workflows
- (If Necessary) Debug Workflows
Detailed instructions for each of the seven simple steps required are given in the following sections.
Step One: Assemble keys.yaml and config.yaml
You can either create new keys.yaml and config.yaml for your pipelines or you can re-use the same keys.yaml and config.yaml that you created for your DevBox.
The risk is that you must ensure that the keys.yaml and config.yaml values are only being used by one process at a time. If two processes run acm commands at the same time and if both processes use the same keys.yaml and config.yaml, you risk conflict errors that would cause one or both processes to fail.
But if only one developer is using the keys.yaml and config.yaml during evaluation of Agile Cloud Manager, and if that developer never runs DevBox acm commands and pipeline acm commands at the same time, then you can save time by re-using the same keys.yaml and config.yaml that you already created.
For teams of developers, and for compliance, it is a good practice to create a separate set of keys.yaml and config.yaml for each stage in the pipeline and for the DevBox.
To create a separate, new keys.yaml and config.yaml for this pipeline tutorial, you would make the following two changes when you do a second run-through of the instructions at this link.
In Section One, Step Nine of these instructions, change the values of appRegistrationName and clientSecretName to read as follows:
python acmDemoSetup.py create azurefiles subscriptionId=valid-subscription-id-guid subscriptionName=validSubscriptionName tenantId=valid-active-directory-tenant-id appRegistrationName=myappregpipe clientSecretName=mycertpipe clientSecretEndDate=2024-12-31
In Section Five, Step Seven of these instructions, change the values of userName, groupName, and keyPairName to read as follows:
python3 acmDemoSetup.py create aws userName=ACMUser_pipe groupName=SuperUserACM_pipe keyPairName=ACMKeyPair_pipe
Except for these changes to these two commands, you can otherwise follow the rest of the instructions identically.
Note that you can make as many versions of keys.yaml and config.yaml as you want, but you must change the values for appRegistrationName, clientSecretName, userName, groupName, and keyPairName for each version similarly to what you just did in the preceding paragraphs.
When you have a keys.yaml file and config.yaml that you are ready to use in a pipeline, you will then be ready to proceed to step two as follows.
Step Two: Open Cloud Provider Dashboards To Monitor Demos
First, open portal.azure.com in another browser window and navigate to the “Resource Groups” page so that you will be able to see all the resources being created and destroyed when you run the demos. You can get to the Resource Groups page by clicking on the Resource Groups icon from the main Azure Portal page as shown in the following screen shot.
Second, (Optional) you can isolate only the one resource group that you created for this Demo by using the filter tool on the “Resource Groups” page as shown in the following screen shot.
Third, open aws.amazon.com and navigate to the cloudformation page so that you can see the cloudformation stacks being created and destroyed when you run the demos. You can navigate to the cloudformation page by typing “cloudformation” in the search tool and then clicking the resulting “cloudformation” link in the search results, as shown in the following screen shot.
Fourth, on the resulting CloudFormation page, click the “Stacks” button at left to open up the Stacks dashboard as shown in the following screen shot. If you have many pre-existing stacks in your account, you can filter by date created to isolate the newest ones.
Step Three: Fork The GitHub Demos Repository
First, log in to your GitHub account.
Second, navigate your browser to https://github.com/AgileCloudInstitute/acm-demos-github , then click the “Fork” button and then the “Create a new fork” button as shown in the following screen shot:
Third, on the resulting screen, keep all the default options, and click on the “Create fork” option.
Step Four: Put the config.yaml and keys.yaml values into two new multi-line GitHub secrets
First, after your fork has been created, click on the “Settings” tab of your forked repository as shown in the following screen shot:
Second, on the resulting “Settings” page of your forked repository, scroll down in the left-hand table-of-contents, click to expand the “Secrets and Variables” option, and then click to select the “Actions” option as shown in the following screen shot.
Third, on the resulting “Actions Secrets and Variables” page, click on the “New repository secret” button to begin the process of entering the contents of your config.yaml as shown in the following screen shot.
Fourth, on the resulting screen, name your new secret “ACM_CONFIG_ONE” and then copy the contents of your config.yaml into the body of the new “ACM_CONFIG_ONE” secret. Make sure that the contents of the “ACM_CONFIG_ONE” are exactly the same as the contents of your config.yaml by simply selecting the entire contents of config.yaml, copying the entire contents into your clipboard, and then pasting the entire contents into the new secret. There should be 31 lines of yaml key/value pairs that were created by automation. Do not change anything from the contents of config.yaml after pasting them in their exact same form. Click the “Add secret” button to save the new secret.
Fifth, click the “New Repository secret” again to create a second secret named “ACM_SECRET_ONE”. Then copy the contents of your keys.yaml file into this new “ACM_SECRET_ONE” secret. Again, make sure that the contents of your “ACM_SECRET_ONE” secrets are identical to the contents of your keys.yaml file. There should be 7 lines of yaml key/value pairs that were created by automation, remembering that you pasted in the AWS values in an earlier step. All values in keys.yaml and in “ACM_SECRET_ONE” should look like the valid values that they need to be. Do not change anything from the contents of keys.yaml after pasting them in their exact same form. Click the “Add secret” button to save the new secret.
Step Five: Run all 4 of the demo workflows
First, click on the “Actions” tab of the forked repository, then click on the “I understand my workflows, go ahead and enable them” button as shown in the following screen shot. Note that you will be creating and destroying cloud resources with the workflows. You can read the workflow code to understand how. And later, when you actually run the workflows later, you will need to have each cloud portal graphical user interface open while the workflows are running, so that you can see what is being created and destroyed, and so that you can clean up if necessary.
Second, on the resulting screen, the workflows will be listed in the left-hand navigation bar.
linux-cloudformation
linux-terraform-packer
windows-arm
windows-custom
Third, click on each workflow one at a time to open the detail page for each workflow. On the detail page for each workflow, click on the “Run workflow” button and then on the next “Run workflow” button that will appear in the drop-down modal. As shown in the following screen shot:
Step Six: Monitor Workflows
After you have clicked to run each workflow separately, then monitor, and if necessary, debug the workflows as they are running, as follows:
First, click on the “All workflows” link on the left-hand navigation panel, as shown in the next screen shot. This will open the resulting “All workflows” detail page, also shown in the following screen shot:
Watch and wait until each of the workflows turns green to indicate that it has completed successfully. But navigate between the cloud provider portals and the GitHub detail pages while the demos are running as described in the next two paragraphs.
Second, adjust the locations of the two other windows that each show your Azure Resource Groups and CloudFormation Stacks, respectively. You can examine the items being created and destroyed by looking a the “Resource Groups” page you opened in Azure in an earlier step above, and by looking at the “Cloud Formation Stacks” page you opened in AWS in an earlier step above. Note that you might need to refresh the Azure Resource Groups page frequently.
Third, you can also click on each of the workflows in GitHub and drill down to the individual jobs and steps to watch the logs being created while the workflows are running.
Step Seven: (If Necessary) Debug Workflows
If you encounter any problems, you can interpret the logs to assess the root cause. Instructions for interpreting the logs are given in the article “Logging for the Agile Cloud Manager” in this website.
The workflows have been tested with thousands and runs, so that the only problems you are likely to encounter might be the result of:
| Possible Cause | Solution |
|---|---|
| GitHub network latency that might occassionally cause a GitHub runner to fail its provisioning process. | If you encounter this problem, simply use the GitHub GUI to re-run the job. |
| Azure or AWS network latency that might cause one of the Agile Cloud Manager’s processes to time out before completion. | If this happens, simply use the GitHub GUI to re-run the job. |
| You might not have assigned proper roles in the Azure Portal GUI | Make sure that the Azure Portal GUI shows that following roles have been assigned to the app registration name that the automation created: Subscription Owner role, Active Directory Global Administrator role, and Active Directory Application Administrator role. |
| Your credentials from keys.yaml or config.yaml might not be valid. | We have controlled for this by automating the process of creating keys.yaml and config.yaml. But you might have made the mistake of modifying either keys.yaml or config.yaml manually. Or you might have introduced an error while copying the contents of keys.yaml or config.yaml into the appropriate GitHub secret. You could try checking your backed up copies of keys.yaml and config.yaml for any obvious errors and then re-pasting the values into the appropriate GitHub Secrets before re-running the affected workflow(s). If that does not work, you could delete and re-create everything from scratch, first by deleting any Azure Resource Groups and AWS CloudFormation stacks that were created by the automation, then following the instructions to delete the IAM resources in the article entitled “Delete IAM Resources”, using the “rm” command to delete any old versions of keys.yaml and config.yaml you might find by running “ls -al” in the cloud shell, and then re-creating new IAM resources with new config.yaml and new keys.yaml by repeating the steps from the article entitled “Assemble keys.yaml and config.yaml”, and then putting the newly re-created keys.yaml and config.yaml values into the appropriate GitHub multiline secrets, and finally re-running the demos to make sure you did each step correctly. |
If you encounter an error that you do not easily understand, then create an issue at this repository with all the steps you took and the complete error message, and we will try to help you.
If all else fails, you can delete any cloud resources as follows:
In Azure, delete any Resource Groups that were created in the new subscription. Deletion of the Resource Groups should cascade delete everything in the Resource Groups.
In AWS, navigate to the CloudFormation page and delete any Stack that was created during this demo. Deletion of the stacks should cascade delete anything that was created in the stacks.
When you no longer need the credentials that were created for this demo, you can delete those credentials using the instructions in the article entitled “Delete IAM Resources”.