Senior Data Engineer
Core Capabilities
- Batch Pipeline Orchestration - Design and implement production-ready ETL/ELT pipelines with Airflow, intelligent dependency resolution, retry logic, and comprehensive monitoring
- Real-Time Streaming - Build event-driven streaming pipelines with Kafka, Flink, Kinesis, and Spark Streaming with exactly-once semantics and sub-second latency
- Data Quality Management - Comprehensive batch and streaming data quality validation covering completeness, accuracy, consistency, timeliness, and validity
- Streaming Quality Monitoring - Track consumer lag, data freshness, schema drift, throughput, and dead letter queue rates for streaming pipelines
- Performance Optimization - Analyze and optimize pipeline performance with query optimization, Spark tuning, and cost analysis recommendations
Key Workflows
Workflow 1: Build ETL Pipeline
Time: 2-4 hours
Steps:
- Design pipeline architecture using Lambda, Kappa, or Medallion pattern
- Configure YAML pipeline definition with sources, transformations, targets
- Generate Airflow DAG with
pipeline_orchestrator.py
- Define data quality validation rules
- Deploy and configure monitoring/alerting
Expected Output: Production-ready ETL pipeline with 99%+ success rate, automated quality checks, and comprehensive monitoring
Workflow 2: Build Real-Time Streaming Pipeline
Time: 3-5 days
Steps:
- Select streaming architecture (Kappa vs Lambda) based on requirements
- Configure streaming pipeline YAML (sources, processing, sinks, quality)
- Generate Kafka configurations with
kafka_config_generator.py
- Generate Flink/Spark job scaffolding with
stream_processor.py
- Deploy and monitor with
streaming_quality_validator.py
Expected Output: Streaming pipeline processing 10K+ events/sec with P99 latency < 1s, exactly-once delivery, and real-time quality monitoring
World-class data engineering for production-grade data systems, scalable pipelines, and enterprise data platforms.
Overview
This skill provides comprehensive expertise in data engineering fundamentals through advanced production patterns. From designing medallion architectures to implementing real-time streaming pipelines, it covers the full spectrum of modern data engineering including ETL/ELT design, data quality frameworks, pipeline orchestration, and DataOps practices.
What This Skill Provides:
- Production-ready pipeline templates (Airflow, Spark, dbt)
- Comprehensive data quality validation framework
- Performance optimization and cost analysis tools
- Data architecture patterns (Lambda, Kappa, Medallion)
- Complete DataOps CI/CD workflows
Best For:
- Building scalable data pipelines for enterprise systems
- Implementing data quality and governance frameworks
- Optimizing ETL performance and cloud costs
- Designing modern data architectures (lake, warehouse, lakehouse)
- Production ML/AI data infrastructure
Quick Start
Pipeline Orchestration
# Generate Airflow DAG from configuration
python scripts/pipeline_orchestrator.py --config pipeline_config.yaml --output dags/
# Validate pipeline configuration
python scripts/pipeline_orchestrator.py --config pipeline_config.yaml --validate
# Use incremental load template
python scripts/pipeline_orchestrator.py --template incremental --output dags/
Data Quality Validation
# Validate CSV file with quality checks
python scripts/data_quality_validator.py --input data/sales.csv --output report.html
# Validate database table with custom rules
python scripts/data_quality_validator.py \
--connection postgresql://user:pass@host/db \
--table sales_transactions \
--rules rules/sales_validation.yaml \
--threshold 0.95
Performance Optimization
# Analyze pipeline performance and get recommendations
python scripts/etl_performance_optimizer.py \
--airflow-db postgresql://host/airflow \
--dag-id sales_etl_pipeline \
--days 30 \
--optimize
# Analyze Spark job performance
python scripts/etl_performance_optimizer.py \
--spark-history-server http://spark-history:18080 \
--app-id app-20250115-001
Real-Time Streaming
# Validate streaming pipeline configuration
python scripts/stream_processor.py --config streaming_config.yaml --validate
# Generate Kafka topic and client configurations
python scripts/kafka_config_generator.py \
--topic user-events \
--partitions 12 \
--replication 3 \
--output kafka/topics/
# Generate exactly-once producer configuration
python scripts/kafka_config_generator.py \
--producer \
--profile exactly-once \
--output kafka/producer.properties
# Generate Flink job scaffolding
python scripts/stream_processor.py \
--config streaming_config.yaml \
--mode flink \
--generate \
--output flink-jobs/
# Monitor streaming quality
python scripts/streaming_quality_validator.py \
--lag --consumer-group events-processor --threshold 10000 \
--freshness --topic processed-events --max-latency-ms 5000 \
--output streaming-health-report.html
Core Workflows
1. Building Production Data Pipelines
Steps:
- Design Architecture: Choose pattern (Lambda, Kappa, Medallion) based on requirements
- Configure Pipeline: Create YAML configuration with sources, transformations, targets
- Generate DAG:
python scripts/pipeline_orchestrator.py --config config.yaml
- Add Quality Checks: Define validation rules for data quality
- Deploy & Monitor: Deploy to Airflow, configure alerts, track metrics
Pipeline Patterns: See frameworks.md for Lambda Architecture, Kappa Architecture, Medallion Architecture (Bronze/Silver/Gold), and Microservices Data patterns.
Templates: See templates.md for complete Airflow DAG templates, Spark job templates, dbt models, and Docker configurations.
2. Data Quality Management
Steps:
- Define Rules: Create validation rules covering completeness, accuracy, consistency
- Run Validation:
python scripts/data_quality_validator.py --rules rules.yaml
- Review Results: Analyze quality scores and failed checks
- Integrate CI/CD: Add validation to pipeline deployment process
- Monitor Trends: Track quality scores over time
Quality Framework: See frameworks.md for complete Data Quality Framework covering all dimensions (completeness, accuracy, consistency, timeliness, validity).
Validation Templates: See templates.md for validation configuration examples and Python API usage.
3. Data Modeling & Transformation
Steps:
- Choose Modeling Approach: Dimensional (Kimball), Data Vault 2.0, or One Big Table
- Design Schema: Define fact tables, dimensions, and relationships
- Implement with dbt: Create staging, intermediate, and mart models
- Handle SCD: Implement slowly changing dimension logic (Type 1/2/3)
- Test & Deploy: Run dbt tests, generate documentation, deploy
Modeling Patterns: See frameworks.md for Dimensional Modeling (Kimball), Data Vault 2.0, One Big Table (OBT), and SCD implementations.
dbt Templates: See templates.md for complete dbt model templates including staging, intermediate, fact tables, and SCD Type 2 logic.
4. Performance Optimization
Steps:
- Profile Pipeline: Run performance analyzer on recent pipeline executions
- Identify Bottlenecks: Review execution time breakdown and slow tasks
- Apply Optimizations: Implement recommendations (partitioning, indexing, batching)
- Tune Spark Jobs: Optimize memory, parallelism, and shuffle settings
- Measure Impact: Compare before/after metrics, track cost savings
Optimization Strategies: See frameworks.md for performance best practices including partitioning strategies, query optimization, and Spark tuning.
Analysis Tools: See tools.md for complete documentation on etl_performance_optimizer.py with query analysis and Spark tuning.
5. Building Real-Time Streaming Pipelines
Steps:
- Architecture Selection: Choose Kappa (streaming-only) or Lambda (batch + streaming) architecture
- Configure Pipeline: Create YAML config with sources, processing engine, sinks, quality thresholds
- Generate Kafka Configs:
python scripts/kafka_config_generator.py --topic events --partitions 12
- Generate Job Scaffolding:
python scripts/stream_processor.py --mode flink --generate
- Deploy Infrastructure: Use Docker Compose for local dev, Kubernetes for production
- Monitor Quality:
python scripts/streaming_quality_validator.py --lag --freshness --throughput
Streaming Patterns: See frameworks.md for stateful processing, stream joins, windowing, exactly-once semantics, and CDC patterns.
Templates: See templates.md for Flink DataStream jobs, Kafka Streams applications, PyFlink templates, and Docker Compose configurations.
Python Tools
pipeline_orchestrator.py
Automated Airflow DAG generation with intelligent dependency resolution and monitoring.
Key Features:
- Generate production-ready DAGs from YAML configuration
- Automatic task dependency resolution
- Built-in retry logic and error handling
- Multi-source support (PostgreSQL, S3, BigQuery, Snowflake)
- Integrated quality checks and alerting
Usage:
# Basic DAG generation
python scripts/pipeline_orchestrator.py --config pipeline_config.yaml --output dags/
# With validation
python scripts/pipeline_orchestrator.py --config config.yaml --validate
# From template
python scripts/pipeline_orchestrator.py --template incremental --output dags/
Complete Documentation: See tools.md for full configuration options, templates, and integration examples.
data_quality_validator.py
Comprehensive data quality validation framework with automated checks and reporting.
Capabilities:
- Multi-dimensional validation (completeness, accuracy, consistency, timeliness, validity)
- Great Expectations integration
- Custom business rule validation
- HTML/PDF report generation
- Anomaly detection
- Historical trend tracking
Usage:
# Validate with custom rules
python scripts/data_quality_validator.py \
--input data/sales.csv \
--rules rules/sales_validation.yaml \
--output report.html
# Database table validation
python scripts/data_quality_validator.py \
--connection postgresql://host/db \
--table sales_transactions \
--threshold 0.95
Complete Documentation: See tools.md for rule configuration, API usage, and integration patterns.
etl_performance_optimizer.py
Pipeline performance analysis with actionable optimization recommendations.
Capabilities:
- Airflow DAG execution profiling
- Bottleneck detection and analysis
- SQL query optimization suggestions
- Spark job tuning recommendations
- Cost analysis and optimization
- Historical performance trending
Usage:
# Analyze Airflow DAG
python scripts/etl_performance_optimizer.py \
--airflow-db postgresql://host/airflow \
--dag-id sales_etl_pipeline \
--days 30 \
--optimize
# Spark job analysis
python scripts/etl_performance_optimizer.py \
--spark-history-server http://spark-history:18080 \
--app-id app-20250115-001
Complete Documentation: See tools.md for profiling options, optimization strategies, and cost analysis.
stream_processor.py
Streaming pipeline configuration generator and validator for Kafka, Flink, and Kinesis.
Capabilities:
- Multi-platform support (Kafka, Flink, Kinesis, Spark Streaming)
- Configuration validation with best practice checks
- Flink/Spark job scaffolding generation
- Kafka topic configuration generation
- Docker Compose for local streaming stacks
- Exactly-once semantics configuration
Usage:
# Validate configuration
python scripts/stream_processor.py --config streaming_config.yaml --validate
# Generate Kafka configurations
python scripts/stream_processor.py --config streaming_config.yaml --mode kafka --generate
# Generate Flink job scaffolding
python scripts/stream_processor.py --config streaming_config.yaml --mode flink --generate --output flink-jobs/
# Generate Docker Compose for local development
python scripts/stream_processor.py --config streaming_config.yaml --mode docker --generate
Complete Documentation: See tools.md for configuration format, validation checks, and generated outputs.
streaming_quality_validator.py
Real-time streaming data quality monitoring with comprehensive health scoring.
Capabilities:
- Consumer lag monitoring with thresholds
- Data freshness validation (P50/P95/P99 latency)
- Schema drift detection
- Throughput analysis (events/sec, bytes/sec)
- Dead letter queue rate monitoring
- Overall quality scoring with recommendations
- Prometheus metrics export
Usage:
# Monitor consumer lag
python scripts/streaming_quality_validator.py \
--lag --consumer-group events-processor --threshold 10000
# Monitor data freshness
python scripts/streaming_quality_validator.py \
--freshness --topic processed-events --max-latency-ms 5000
# Full quality validation
python scripts/streaming_quality_validator.py \
--lag --freshness --throughput --dlq \
--output streaming-health-report.html
Complete Documentation: See tools.md for all monitoring dimensions and integration patterns.
kafka_config_generator.py
Production-grade Kafka configuration generator with performance and security profiles.
Capabilities:
- Topic configuration (partitions, replication, retention, compaction)
- Producer profiles (high-throughput, exactly-once, low-latency, ordered)
- Consumer profiles (exactly-once, high-throughput, batch)
- Kafka Streams configuration with state store tuning
- Security configuration (SASL-PLAIN, SASL-SCRAM, mTLS)
- Kafka Connect source/sink configurations
- Multiple output formats (properties, YAML, JSON)
Usage:
# Generate topic configuration
python scripts/kafka_config_generator.py \
--topic user-events --partitions 12 --replication 3 --retention-hours 168
# Generate exactly-once producer
python scripts/kafka_config_generator.py \
--producer --profile exactly-once --transactional-id producer-001
# Generate Kafka Streams config
python scripts/kafka_config_generator.py \
--streams --application-id events-processor --exactly-once
Complete Documentation: See tools.md for all profiles, security options, and Connect configurations.
Reference Documentation
Comprehensive data engineering frameworks and patterns:
- Architecture Patterns: Lambda, Kappa, Medallion, Microservices data architecture
- Data Modeling: Dimensional (Kimball), Data Vault 2.0, One Big Table
- ETL/ELT Patterns: Full load, incremental load, CDC, SCD, idempotent pipelines
- Data Quality: Complete framework covering all quality dimensions
- DataOps: CI/CD for data pipelines, testing strategies, monitoring
- Orchestration: Airflow DAG patterns, backfill strategies
- Real-Time Streaming: Stateful processing, stream joins, windowing strategies, exactly-once semantics, event time processing, watermarks, backpressure, Apache Flink patterns, AWS Kinesis patterns, CDC for streaming
- Governance: Data catalog, lineage tracking, access control
Production-ready code templates and examples:
- Airflow DAGs: Complete ETL DAG, incremental load, dynamic task generation
- Spark Jobs: Batch processing, streaming, optimized configurations
- dbt Models: Staging, intermediate, fact tables, dimensions with SCD Type 2
- SQL Patterns: Incremental merge (upsert), deduplication, date spine, window functions
- Python Pipelines: Data quality validation class, retry decorators, error handling
- Real-Time Streaming: Apache Flink DataStream jobs (Java), Kafka Streams applications, PyFlink jobs, AWS Kinesis consumers, Docker Compose for streaming stack
- Kafka Configs: Producer/consumer properties templates, topic configurations, security configurations
- Docker: Dockerfiles for data pipelines, Docker Compose for local development including streaming stack (Kafka, Flink, Schema Registry)
- Configuration: dbt project config, Spark configuration, Airflow variables, streaming pipeline YAML
- Testing: pytest fixtures, integration tests, data quality tests
Python automation tool documentation:
- pipeline_orchestrator.py: Complete usage guide, configuration format, DAG templates
- data_quality_validator.py: Validation rules, dimension checks, Great Expectations integration
- etl_performance_optimizer.py: Performance analysis, query optimization, Spark tuning
- stream_processor.py: Streaming pipeline configuration, validation, job scaffolding generation
- streaming_quality_validator.py: Consumer lag, data freshness, schema drift, throughput monitoring
- kafka_config_generator.py: Topic, producer, consumer, Kafka Streams, and Connect configurations
- Integration Patterns: Airflow, dbt, CI/CD, monitoring systems, Prometheus
- Best Practices: Configuration management, error handling, performance, monitoring, streaming quality
Tech Stack
Core Technologies:
- Languages: Python 3.8+, SQL, Scala (Spark), Java (Flink)
- Orchestration: Apache Airflow, Prefect, Dagster
- Batch Processing: Apache Spark, dbt, Pandas
- Stream Processing: Apache Kafka, Apache Flink, Kafka Streams, Spark Structured Streaming, AWS Kinesis
- Storage: PostgreSQL, BigQuery, Snowflake, Redshift, S3, GCS
- Schema Management: Confluent Schema Registry, AWS Glue Schema Registry
- Containerization: Docker, Kubernetes
- Monitoring: Datadog, Prometheus, Grafana, Kafka UI
Data Platforms:
- Cloud Data Warehouses: Snowflake, BigQuery, Redshift
- Data Lakes: Delta Lake, Apache Iceberg, Apache Hudi
- Streaming Platforms: Apache Kafka, AWS Kinesis, Google Pub/Sub, Azure Event Hubs
- Stream Processing Engines: Apache Flink, Kafka Streams, Spark Structured Streaming
- Workflow: Airflow, Prefect, Dagster
Integration Points
This skill integrates with:
- Orchestration: Airflow, Prefect, Dagster for workflow management
- Transformation: dbt for SQL transformations and testing
- Quality: Great Expectations for data validation
- Monitoring: Datadog, Prometheus for pipeline monitoring
- BI Tools: Looker, Tableau, Power BI for analytics
- ML Platforms: MLflow, Kubeflow for ML pipeline integration
- Version Control: Git for pipeline code and configuration
See tools.md for detailed integration patterns and examples.
Best Practices
Pipeline Design:
- Idempotent operations for safe reruns
- Incremental processing where possible
- Clear data lineage and documentation
- Comprehensive error handling
- Automated recovery mechanisms
Data Quality:
- Define quality rules early
- Validate at every pipeline stage
- Automate quality monitoring
- Track quality trends over time
- Block bad data from downstream
Performance:
- Partition large tables by date/region
- Use columnar formats (Parquet, ORC)
- Leverage predicate pushdown
- Optimize for your query patterns
- Monitor and tune regularly
Operations:
- Version control everything
- Automate testing and deployment
- Implement comprehensive monitoring
- Document runbooks for incidents
- Regular performance reviews
Performance Targets
Batch Pipeline Execution:
- P50 latency: < 5 minutes (hourly pipelines)
- P95 latency: < 15 minutes
- Success rate: > 99%
- Data freshness: < 1 hour behind source
Streaming Pipeline Execution:
- Throughput: 10K+ events/second sustained
- End-to-end latency: P99 < 1 second
- Consumer lag: < 10K records behind
- Exactly-once delivery: Zero duplicates or losses
Data Quality (Batch):
- Quality score: > 95%
- Completeness: > 99%
- Timeliness: < 2 hours data lag
- Zero critical failures
Streaming Quality:
- Data freshness: P95 < 5 minutes from event generation
- Late data rate: < 5% outside watermark window
- Dead letter queue rate: < 1%
- Schema compatibility: 100% backward/forward compatible changes
Cost Efficiency:
- Cost per GB processed: < $0.10
- Cloud cost trend: Stable or decreasing
- Resource utilization: > 70%
Resources
- Frameworks Guide: references/frameworks.md
- Code Templates: references/templates.md
- Tool Documentation: references/tools.md
- Python Scripts:
scripts/ directory
Version: 2.0.0
Last Updated: December 16, 2025
Documentation Structure: Progressive disclosure with comprehensive references
Streaming Enhancement: Task #8 - Real-time streaming capabilities added
