Data Engineering Training in Perungudi

• Introduction to the Modern Data Stack and Data Lifecycle
• Understanding Batch vs. Real-Time Data Processing
• Core Data Architecture Patterns (Lambda, Kappa, Data Mesh basics)
• Data Modeling Fundamentals (Normalization vs. Denormalization)
• Introduction to Data Quality, Governance, and Security

Beginner Design and document a scalable conceptual data architecture for a mock e-commerce company.

• Advanced SQL: Window Functions, CTEs, and Complex Joins
• Query Optimization, Indexing, and Execution Plans
• Dimensional Data Modeling: Star and Snowflake Schemas
• Handling Slowly Changing Dimensions (SCD Types 1, 2, and 3)
• Introduction to Modern Cloud Data Warehouses (Snowflake/BigQuery)

Beginner to Intermediate Build an optimized, multi-layer data warehouse schema (Bronze, Silver, Gold) using advanced SQL.

• Python Fundamentals: Data Structures, Functions, and Error Handling
• Advanced Data Manipulation with Pandas and Polars
• Working with APIs, JSON, and Web Scraping for Data Ingestion
• Handling Modern File Formats: Parquet, Avro, and ORC
• Writing Clean, Modular, and Automated Python Scripts for Data Tasks

Intermediate Automate a Python script extracting REST API data, transforming it, and saving as Parquet.

• Cloud Fundamentals: IAM, VPCs, and Security Best Practices
• Cloud Storage Solutions: Data Lakes (S3, ADLS, GCS)
• Serverless Compute and Event-Driven Data Triggers (Lambda/Functions)
• Managed Database Services vs. Cloud Data Warehouses
• Introduction to Cloud Infrastructure as Code (Terraform basics)

Intermediate Deploy a secure, multi-tier cloud data lake infrastructure using Infrastructure as Code (IaC).

• Apache Spark Architecture: Drivers, Executors, and Clusters
• PySpark DataFrames: Transformations, Actions, and Spark SQL
• Handling Large-Scale Joins, Aggregations, and Window Functions
• Advanced Performance Tuning: Partitioning, Bucketing, and Caching
• Debugging Spark: Understanding the UI and resolving Data Skew

Advanced Process a large messy dataset using PySpark, optimizing job performance 3x faster via tuning.

• The Evolution from Data Lake to Data Lakehouse
• Deep Dive into Apache Iceberg and Delta Lake
• Implementing ACID Transactions on Object Storage
• Time Travel, Schema Evolution, and Partition Evolution
• Integrating Lakehouse formats with Spark and Flink

Advanced Build a production-grade Data Lakehouse with Apache Iceberg, implementing time-travel and schema evolution.

• The Shift from ETL to ELT in Modern Cloud Warehouses
• dbt Fundamentals: Models, Sources, and Materializations
• Advanced dbt: Jinja Templating, Macros, and Custom Tests
• Building Incremental Models and Handling Late-Arriving Data
• Data Lineage, Documentation, and dbt Cloud integration

Advanced Build an end-to-end ELT pipeline using dbt with automated data quality tests and documentation.

• Event-Driven Architecture and the Pub/Sub Model
• Apache Kafka Core Concepts: Topics, Partitions, Brokers, and Consumers
• Building Producers and Consumers with Python/Java
• Kafka Connect and Kafka Streams for Data Integration
• Introduction to Apache Flink for Stateful Stream Processing

Advanced Build a real-time clickstream analytics pipeline ingesting live events via Kafka and aggregating via Flink.

• DataOps Principles: Version Control (Git) and Branching Strategies
• Containerizing Data Apps with Docker and Kubernetes basics
• Building CI/CD Pipelines for Data (GitHub Actions/GitLab CI)
• Automated Data Quality Monitoring with Great Expectations
• Cloud FinOps: Monitoring, Alerting, and Optimizing Compute Costs

Expert Implement an automated CI/CD pipeline with data quality checks and a cloud cost-monitoring dashboard.

• Understanding the Data Requirements for GenAI and LLMs
• Processing Unstructured Data: Text, PDFs, Images, and Audio
• Introduction to Vector Databases (Pinecone, Milvus, pgvector)
• Building Data Pipelines for RAG (Retrieval-Augmented Generation)
• Feature Stores and ML Data Pipeline Automation

Expert Build an AI-ready pipeline ingesting unstructured documents, creating embeddings, and storing them in a Vector DB.

• Principles of Data Mesh: Domain Ownership and Data as a Product
• Data Fabric Concepts and Metadata Management
• Implementing Data Contracts between Producers and Consumers
• Master Data Management (MDM) and Reference Data
• Advanced Data Governance, Cataloging (DataHub), and PII Masking

Expert Design a comprehensive Data Mesh architecture for a global enterprise, including data contracts and catalogs.

• End-to-End System Integration: Batch, Streaming, and AI pipelines
• Production Deployment, Monitoring, and Alerting Strategies
• Disaster Recovery, Backup, and High Availability Planning
• Portfolio Building: Documenting Architecture on GitHub and Medium
• Mock Interviews, Resume Review, and Career Placement Prep

Expert Deploy a complete Enterprise Data & AI Platform combining batch, streaming, dbt, and AI vector pipelines.