15 Topics
10 Days
75 Sessions
Programme Overview
Training Description
Who Should Attend
- Data Architects
- Data Engineers
- Big Data Engineers
- Cloud Data Engineers
- Data Platform Engineers
- Data Lake Engineers
- Enterprise Architects
- Cloud Architects
- Analytics Engineers
- Data Warehouse Developers
- Database Administrators
- Database Engineers
- Data Scientists
- Machine Learning Engineers
Session Objectives
- Understand the core concepts of data lakes and their role in a data ecosystem.
- Learn to differentiate between a data lake, data warehouse, and data mart.
- Acquire skills in designing a multi-layered data lake architecture (e.g., Bronze, Silver, Gold).
- Comprehend techniques for ingesting data from various sources into a data lake.
- Explore strategies for processing and transforming data within a data lake.
- Understand the importance of data cataloging, metadata management, and data discovery.
- Gain insights into implementing robust security and access control for data lakes.
- Develop a practical understanding of data governance, quality, and lineage in a data lake environment.
- Master the use of key cloud-native services for data lake implementation.
- Acquire skills in optimizing data lake storage and compute costs.
- Learn to apply best practices for building scalable and reliable data pipelines.
- Comprehend techniques for handling streaming and real-time data ingestion.
- Explore strategies for integrating a data lake with existing data warehouses and BI tools.
- Understand the importance of monitoring, logging, and performance tuning for a data lake.
- Develop the ability to lead and implement a successful Data Lake Architecture and Implementation project.
About the Course
This comprehensive training course equips participants with the knowledge and practical skills required to design, implement, manage, and optimize modern enterprise data lakes. It explores how organizations can build scalable platforms capable of storing and processing massive volumes of structured, semi-structured, and unstructured data from diverse sources.Participants will examine data lake architecture, storage design, ingestion frameworks, data organization, metadata management, data processing, security, governance, quality, and lifecycle management. The course also addresses the integration of data lakes with cloud platforms, data warehouses, lakehouses, business intelligence, advanced analytics, and artificial intelligence environments.Through practical exercises and real-world architecture scenarios, participants will learn how to design scalable data lake environments, develop ingestion pipelines, organize data into logical layers, manage metadata and lineage, implement governance controls, optimize performance and costs, and avoid common data lake challenges such as data silos, poor quality, and uncontrolled data growth.
Curriculum & Topics
15 Topics | 75 Sessions
-
Workshop 1.1:
What is a data lake and its key characteristics?
-
Workshop 1.2:
The evolution from data warehouses to data lakes and lakehouses.
-
Workshop 1.3:
Key components of a data lake: storage, compute, and metadata.
-
Workshop 1.4:
Benefits and challenges of implementing a data lake.
-
Workshop 1.5:
Business drivers for adopting a data lake architecture.
-
Workshop 2.1:
The multi-layered approach: Raw, Staging, Curated, and Consumption layers.
-
Workshop 2.2:
The medallion architecture (Bronze, Silver, Gold) and its purpose.
-
Workshop 2.3:
Choosing the right file formats for data lake storage (e.g., Parquet, Avro, ORC).
-
Workshop 2.4:
Understanding schema-on-read vs. schema-on-write.
-
Workshop 2.5:
Designing for scalability and flexibility.
-
Workshop 3.1:
Introduction to Amazon S3, Azure Data Lake Storage (ADLS), and Google Cloud Storage (GCS).
-
Workshop 3.2:
Managing storage: buckets, containers, and data organization.
-
Workshop 3.3:
Tiered storage strategies and lifecycle management.
-
Workshop 3.4:
Cost management for object storage.
-
Workshop 3.5:
Best practices for file and folder naming conventions.
-
Workshop 4.1:
Batch vs. Streaming data ingestion.
-
Workshop 4.2:
Tools for data ingestion: AWS Glue, Azure Data Factory, Google Cloud Dataflow.
-
Workshop 4.3:
Ingesting structured data from databases (CDC, bulk loading).
-
Workshop 4.4:
Ingesting semi-structured data (JSON, XML) and unstructured data.
-
Workshop 4.5:
Data ingestion security and pipeline monitoring.
-
Workshop 5.1:
The role of Spark, Databricks, and other processing engines.
-
Workshop 5.2:
Using SQL for data transformation (e.g., AWS Athena, Azure Synapse SQL, Google BigQuery).
-
Workshop 5.3:
Building ETL/ELT pipelines for data cleansing and enrichment.
-
Workshop 5.4:
The importance of idempotency and fault tolerance in processing jobs.
-
Workshop 5.5:
Orchestrating data pipelines with tools like Airflow or cloud-native services.
-
Workshop 6.1:
What is a data catalog and why is it essential for data lakes?
-
Workshop 6.2:
Metadata management: business, technical, and operational metadata.
-
Workshop 6.3:
Tools for data cataloging: AWS Glue Data Catalog, Azure Purview, Google Data Catalog.
-
Workshop 6.4:
Data governance and ownership with a data catalog.
-
Workshop 6.5:
Enabling self-service analytics and data discovery.
-
Workshop 7.1:
Implementing a secure perimeter for the data lake.
-
Workshop 7.2:
Identity and Access Management (IAM) for data resources.
-
Workshop 7.3:
Data encryption at rest and in transit.
-
Workshop 7.4:
Role-based access control (RBAC) and attribute-based access control (ABAC).
-
Workshop 7.5:
Auditing, logging, and monitoring access to data.
-
Workshop 8.1:
Establishing a data governance framework for a data lake.
-
Workshop 8.2:
Data quality rules and validation at each layer.
-
Workshop 8.3:
Data lineage and tracking data flow.
-
Workshop 8.4:
Managing data lifecycle and retention policies.
-
Workshop 8.5:
Creating a single source of truth within the data lake.
-
Workshop 9.1:
Deep dive into AWS S3, AWS Glue, and AWS Lake Formation.
-
Workshop 9.2:
Using AWS Athena and Redshift Spectrum for querying the data lake.
-
Workshop 9.3:
Building a real-world data lake on AWS.
-
Workshop 9.4:
Best practices for performance and cost optimization on AWS.
-
Workshop 9.5:
Case studies of AWS data lake implementations.
-
Workshop 10.1:
Deep dive into Azure Data Lake Storage (ADLS) and Azure Synapse Analytics.
-
Workshop 10.2:
Using Azure Data Factory and Databricks for data processing.
-
Workshop 10.3:
Implementing a data lake with Azure Purview for governance.
-
Workshop 10.4:
Best practices for security and cost management on Azure.
-
Workshop 10.5:
Case studies of Azure data lake implementations.
-
Workshop 11.1:
Deep dive into Google Cloud Storage (GCS) and Google Cloud Dataproc.
-
Workshop 11.2:
Using BigQuery for querying and analytics on the data lake.
-
Workshop 11.3:
Implementing data ingestion with Google Cloud Dataflow and Pub/Sub.
-
Workshop 11.4:
Best practices for performance and cost optimization on GCP.
-
Workshop 11.5:
Case studies of GCP data lake implementations.
-
Workshop 12.1:
Introduction to the lakehouse architecture and its advantages.
-
Workshop 12.2:
The role of Delta Lake in creating a lakehouse.
-
Workshop 12.3:
Using Databricks on any cloud to build a lakehouse.
-
Workshop 12.4:
Unifying data warehousing and data lakes.
-
Workshop 12.5:
Migrating from a traditional data warehouse to a lakehouse.
-
Workshop 13.1:
Architecting for real-time data ingestion.
-
Workshop 13.2:
Using stream processing engines (e.g., Apache Flink, Spark Streaming).
-
Workshop 13.3:
Cloud services for streaming: Kinesis, Event Hubs, Pub/Sub.
-
Workshop 13.4:
Processing and storing real-time data in a data lake.
-
Workshop 13.5:
Building a lambda or kappa architecture for hybrid data.
-
Workshop 14.1:
Monitoring data pipelines and infrastructure.
-
Workshop 14.2:
Troubleshooting common data lake issues.
-
Workshop 14.3:
Automation of tasks and CI/CD for data lake projects.
-
Workshop 14.4:
Performance tuning: compaction, partitioning, and indexing.
-
Workshop 14.5:
Cost management and resource allocation strategies.
-
Workshop 15.1:
Participants work in teams on a business case to design a data lake.
-
Workshop 15.2:
Exercise: Define the architecture, choose cloud services, and design the data flow.
-
Workshop 15.3:
Create a data ingestion plan and a data quality framework.
-
Workshop 15.4:
Present the final design and justify key decisions.
-
Workshop 15.5:
Discussion on real-world challenges and solutions.
