Data Modeling, Storage, and Pipelines

This unit introduces core data modeling techniques, storage architectures, and data pipeline patterns used in data engineering. Students learn relational modeling (normalization/denormalization), compare databases, warehouses, lakes, and lakehouses, build and orchestrate ETL/streaming pipelines (including a hands-on Airflow lab), and assess data quality issues and remediation strategies.

Learning Objectives

• Analyze relational and non-relational data models and storage architectures (databases, data warehouses, data marts, data lakes, and lakehouses) to recommend the most appropriate solution for given engineering requirements
• Design normalized and denormalized database schemas and apply optimization techniques (indexing, partitioning, and query tuning) to meet performance and scalability requirements
• Apply pipeline design patterns (batch, streaming, and Lambda architecture) to construct and orchestrate ETL/ELT workflows, including implementing and scheduling a basic pipeline using Apache Airflow
• Evaluate data quality problems by identifying common sources of bad data and applying data validation, cleansing, and monitoring strategies across the data lifecycle