Agile Data Lake House Architecture with the Agile Cloud Manager
The entire transcript of this video is given below the video so that you can read and consume it at your own pace. Screen shots of each slide are also given below to make it easier for you to connect the words with the pictures. We recommend that you both read and watch to make it easier to more completely grasp the material.
An entire data lake house can be put under Agile management if you decompose it into meaningful subcomponents that can be deployed individually.
Agile Cloud Manager makes it easy to put a data lake house under Agile management because Agile Cloud Manager makes it easy for you to deploy subcomponents individually or as groups.
Agenda
In this workshop, we will explain how to manage Agile data lake houses by taking you through eight slides summarized as follows.
- First, we will explain what an Agile data lake house is.
- Second, we will describe each of the major components of a data lake house.
- Third, we will give a view of the layers of a data lake house’s architecture.
- Fourth, we will show a way to examine whether each element of a data lake house has a different life cycle than other elements.
- Fifth, we will explain how to group elements of a data lake house into a small number of systems.
- Sixth, we will describe a way to design each system as an organized group of some of the components of your lake house.
- Seventh, we will discuss how to define Agile products for each system that can be put under Agile management.
- Eighth, and finally, we will describe an example Agile data lake house appliance using the Agile Cloud Manager.
What is an Agile Data Lake House?
Now let’s examine what a data lake house is.
An Agile data lake house is a data architecture that combines the following three things:
- A data lake, which is a flexible and scalable place to store unstructured data including things like documents, images, streaming data, and other miscellaneous things that an enterprise can collect in the course of its operations.
- A data warehouse, which is a highly-structured and easily-managed central repository for the types of data that get stored in databases.
- Agile management, which is an iterative way of evolving systems that includes decomposing systems into software-defined functional components, and deploying those components through automation.
An Agile data lake house is a data architecture that combines these best aspects of data lakes, data warehouses, and Agile management.
Components of a Data Lake House
To illustrate the components of a data lake house, let’s begin with the various types of data sources on the left, and then proceed through the various components all the way to the end users of the data lake house, which you can see on the right.
The numbers will guide us through the components.
Number 1 on the slide marks the data sources.
Some of the many types of data that can be ingested into a data lake house include:
- Databases
- Software-As-A-Service, SaaS, data outputs.
- Streaming data from Internet-Of-Things, I.O.T., appliances and other sources.
- Batch files from legacy technologies like mainframes.
- Image files including pictures, and also scanned images of things like handwritten documents.
- Documents including text documents, word processing documents, spreadsheets, and other artifacts of human work.
- Video files.
- Social media data.
- And many other types of data sources.
Storage is marked by number 2 on the slide. The storage layer includes a data lake and a data warehouse.
A central data catalog is number 3 on the slide. The data catalog is a meta-layer that enables all the data in the entire data lake to be organized and searched. Your organization creates its own data catalog as a navigable map to all the resources in both the data lake and the data warehouse. A well-organized data catalog enables you to search the unstructured data in the data lake as easily as you can search the structured data in the data warehouse.
Number 4 on the slide marks the data cleansing that gets done in the lake house. Many of your data pipelines are summarized here. This includes Extract-Transform-Load, E.L.T., pipelines. This also includes Extract-Load-Transform, E.T.L., pipelines.
The transaction layer is marked by number 5 on the slide. The transaction layer is what connects the data catalog with the various applications that end users will employ to connect with the data that is summarized in the data catalog.
Integrations between the lake house and other tools employed by end users are summarized by number 6 on the slide. The types of tools that can integrate include:
- SQL databases
- Business Intelligence tools
- Machine learning tools
- Streaming data consuming applications
Many different types of end users are summarized by number 7 on the slide. The types of users include as many different types of people as there might be in your extended enterprise. These include executives, sales teams, marketing teams, operations teams, human resources teams, financial teams, and other types of users.
Number 8 on the slide illustrates that all these different components of the lake house need to have quality administration, including enterprise-grade security, data integrity, and performance.
Different lake house vendors have their own various tools for managing each of the 8 basic types of components of a data lake house.
Architecture of an Agile, tool-agnostic data lake house begins by looking at all these components together as functional units before you even begin to make decisions about which tools to use for each component.
Layered Architecture
It can also help to look at a data lake house as four layers that include and ingestion layer, a storage layer, a data management layer, and a consumption layer.
The ingestion layer is where many different types of data are ingested into both the data lake and the data warehouse. Different types of automation are employed to do the data ingestion depending on the type of data. For example, streaming data might be ingested using tools like Kafka. Batch files might be ingested by scheduled jobs. Other types of data might be ingested using other types of tools.
The storage layer is where all the many different types of data will be housed. The storage layer can be partitioned so that it will be clear how close each partition is to being ready to be consumed. For example, there might be:
- a raw data partition,
- a trusted-conformed partition,
- an enriched data partition, and
- a modeled data partition.
The data management layer is where all the computing work gets done. This includes running the many pipelines that transform and write data into the different partitions during the work of discovery, cleaning, processing, monitoring, serving, and archiving. The data catalog is part of the data management layer. All the governance and security get done at the data management layer.
The consumption layer is what many different types of user groups use to interact with the curated unified data store. Each of these various types of user groups utilizes its own types of tools based on their skill sets and based on their needs.
Examine The Life Cycle Of Each Element Of Lake House
An Agile data lake house needs to have each of its components managed separately and deployed separately, while remaining part of an integrated, testable, whole.
The architecture therefore must consider the different life cycles of each of the different components.
You can start to identify the different life cycles if you examine three main things:
- The function that is performed by each element.
- The people who are involved with each element.
- The workflows that interact with each element.
Group Elements Of Lake House Into Separate Systems
A data lake house is complex enough that it can include several different systems.
Each system in your data lake house can best be defined in a way that has clear functionality, life cycle, and building blocks.
It can help to think of your data lake appliance as a core system with several satellite systems.
The core system might include things that are shared by all the other systems. For example, your core system might include storage and IAM roles, among other things.
And each satellite system might define a functional unit that interacts directly with the core system to do specific things. A satellite system might correspond with a specific user group that does specific functions that can be grouped together as a coherent whole.
Building blocks refer to the 3rd party infrastructure templates that the Agile Cloud Manager orchestrates.
Organize Components Of Each System
The Agile Cloud Manager’s simple templating language makes it very easy for you to organize the components of each system into units that can be deployed and operated on individually.
Several different types of services might be included in each system.
The types of services can each do very different, but yet complementary things.
There might be multiple different instances of each type of service in a given system. Each instance of each type of service might be instantiated using different parameters, so that each instance of each type of service might be designed to behave differently from the other instances. One example might be if you have different instances of the same type of service running in different regions.
An optional foundation can also be defined for each of your systems.
The foundation is intended to provide shared resources that are used by multiple different types of services. Networking is one example of something that might best be defined in a shared foundation.
The various types of services might each be deployed into the same network, and that same shared network would best be defined in a shared foundation.
Define Agile Products For Each Component
Agile Product Management can enable your organization to manage the evolution of each element of your data lake house separately and as a group of distinct entities, so that your organization can evolve more effectively.
Four layers can organize each of the new Agile products that you can define. These layers include:
- One lake house appliance product that manages the entire appliance.
- Several system products which each manage one of the systems in the appliance.
- Many different service type products, which each manage one of the types of services in a given appliance.
- Many building block products, which each manage one of the third-party templates that gets orchestrated by a service type or by a foundation of a system. These third-party building blocks are written for third party tools like CloudFormation, Terraform, Azure Resource Manager, etc.
Each of your new Agile products can get all the individual attention it needs including:
- Dedicated schedules for creation, update, and deletion.
- Ongoing requirements gathering.
- Architecture that has the benefit of your architectural governance procedures.
- Work items that are managed within your Agile framework.
Example Lake House Appliance
A working example of a software-defined data lake house appliance is available in the marketplace section of AgileCloudInstitute.io. You can seed your Agile data lake house program with this working example.
The example data lake house appliance is composed of three systems:
- A core system, which contains shared elements for your entire lake house.
- A data engineer system, which builds upon the core system, adding items specifically for data engineers.
- A data scientist system, which also builds upon the core system, but which adds items specifically for data scientists.
Each of the systems in the example lake house appliance includes its own:
- Foundation, composed of shared elements.
- Service types, each of which performs specific operations.
Every aspect of the example data lake house appliance is software-defined, meaning that nothing will remain after you destroy each instance of the lake house appliance.
Every level defined in the example lake house appliance can be orchestrated with its own CLI deployments that perform create, update, delete CRUD operations on each level.
You can learn about the CLI commands and about the object model in the Engineering section of AgileCloudInstitute.io.
Next Steps:
The next steps after reading this article include:
- Contact us for technical support or consulting.
- Connect with the Architect of Agile Cloud Manager on LinkedIn to be invited to events and to answer your questions.
- Do the hands-on training for the free example data lake house in the marketplace section of AgileCloud Institute.io
- Do a Proof of Concept POC of Agile Cloud Manager in your organization using the data lake house example appliance as an initial seed to accelerate the start of your project. We can provide consulting and technical support to make your POC experience easier.