Data EngineeringData Engineering
Data engineering course at STMIK Tazkia in two parts: big-data engineering and the cold path (data modelling and columnar formats, large-scale storage and tiering, lakehouse and the medallion architecture, single-machine ETL with Arrow, Polars and DuckDB, orchestration and dbt, storage engines and NoSQL) and the hot path (from Hadoop to Spark at scale, Kafka/NATS/CDC streaming, hot and cold architectures, dashboards and data apps, data security, governance, GRC and observability).Mata kuliah Data Engineering STMIK Tazkia dengan dua capaian: big data engineering dan cold path (pemodelan data dan format columnar, storage skala besar dan tiering, lakehouse dan medallion architecture, ETL di satu mesin dengan Arrow, Polars, dan DuckDB, orchestration dan dbt, storage engine dan NoSQL) serta hot path (dari Hadoop ke Spark skala besar, streaming dengan Kafka/NATS/CDC, arsitektur hot dan cold, dashboard dan data app, data security, governance, GRC, dan observability).
Data Engineering 101Data Engineering 101
Opening meeting: you learn what data engineering is, why data-driven organizations need it, its lifecycle, and the map of this course's two learning outcomes.Pertemuan pembuka: Anda memahami apa itu data engineering, mengapa organisasi data-driven membutuhkannya, siklus hidupnya, serta peta dua capaian mata kuliah ini.
- 📄 Slide Pertemuan 1 v13112026
Data modelling and columnar formatsPemodelan data dan format columnar
You model data for transactions (OLTP) and analytics (OLAP), including dimensional modeling and star schemas, and choose the right storage format, such as columnar formats.Anda memodelkan data untuk transaksi (OLTP) dan analitik (OLAP), termasuk dimensional modeling dan star schema, serta memilih format penyimpanan yang tepat seperti format columnar.
- 📄 Slide Pertemuan 2 v13112026
Large-scale storage and tieringStorage skala besar dan tiering
You design a large-scale storage layer (HDFS, object storage, replication vs erasure coding) and a hot, warm, cold, and archive storage tiering policy.Anda merancang storage layer skala besar (HDFS, object storage, replikasi vs erasure coding) dan kebijakan storage tiering hot, warm, cold, dan archive.
- 📄 Slide Pertemuan 3 v07112026
Lakehouse and the medallion architectureLakehouse dan medallion architecture
You compare data warehouses, data lakes, and lakehouses (Iceberg, Delta Lake, Hudi) and design a bronze, silver, gold medallion architecture.Anda membandingkan data warehouse, data lake, dan lakehouse (Iceberg, Delta Lake, Hudi) serta merancang medallion architecture bronze, silver, gold.
- 📄 Slide Pertemuan 4 v13112026
ETL on a single machine: Arrow, Polars, DuckDBETL di satu mesin: Arrow, Polars, DuckDB
You build high-performance ETL transformations on a single machine (Apache Arrow, Polars, DuckDB) and choose the right engine before moving to a cluster.Anda membangun transformasi ETL berkinerja tinggi di satu mesin (Apache Arrow, Polars, DuckDB) dan memilih engine yang tepat sebelum beralih ke klaster.
- 📄 Slide Pertemuan 5 v07112026
Orchestration, dbt, and data qualityOrchestration, dbt, dan data quality
You orchestrate ETL pipelines with Airflow or Dagster and dbt so they are idempotent, backfillable, and tested for data quality.Anda meng-orchestrate pipeline ETL dengan Airflow atau Dagster dan dbt agar idempotent, dapat di-backfill, dan teruji data quality-nya.
- 📄 Slide Pertemuan 6 v13112026
Storage engines, NoSQL, and servingStorage engine, NoSQL, dan serving
You choose storage engines (B-tree vs LSM-tree) and serving stores (NoSQL, time-series, search) to fit the access pattern, including immutable storage and ledgers.Anda memilih storage engine (B-tree vs LSM-tree) dan serving store (NoSQL, time-series, search) sesuai pola akses, termasuk immutable storage dan ledger.
- 📄 Slide Pertemuan 7 v13112026
Midterm - group project: a bronze-silver-gold ETL pipelineUTS - tugas besar: pipeline ETL bronze-silver-gold
Midterm exam as a major project, "Bronze-silver-gold ETL pipeline" on a single machine for one real dataset: ingestion, cleaning and PII tokenization, a star schema, orchestration, data quality tests, storage tiering, and a simple dashboard.Ujian Tengah Semester berupa tugas besar "Pipeline ETL bronze-silver-gold" di satu mesin untuk satu dataset nyata: ingest, pembersihan dan tokenization PII, star schema, orchestration, data quality test, storage tiering, dan dashboard sederhana.
- 📄 Brief Tugas Besar UTS v07112026
From Hadoop to SparkDari Hadoop ke Spark
You study the Hadoop ecosystem and how Spark works internally, then write Spark jobs in Scala and PySpark.Anda mempelajari ekosistem Hadoop dan cara kerja Spark dari dalam, lalu menulis job Spark dalam Scala dan PySpark.
- 📄 Slide Pertemuan 9 v13112026
Spark at scale: HPC for dataSpark skala besar: HPC data
You run large-scale Spark pipelines efficiently (the Catalyst optimizer, join strategies, partitioning) and prove their scaling with measurements.Anda menjalankan pipeline Spark skala besar secara efisien (optimizer Catalyst, strategi join, partitioning) dan membuktikan scaling-nya dengan pengukuran.
- 📄 Slide Pertemuan 10 v07112026
Kafka, NATS, CDC, and stream processingKafka, NATS, CDC, dan stream processing
You build an event-driven hot path with Kafka, NATS, and CDC, and process data streams in real time with Spark Structured Streaming.Anda membangun hot path berbasis event dengan Kafka, NATS, dan CDC, serta mengolah aliran data secara real-time dengan Spark Structured Streaming.
- 📄 Slide Pertemuan 11 v13112026
Hot and cold architectures in industryArsitektur hot dan cold di industri
You design hot and cold architectures with a serving layer (Lambda vs Kappa, real-time OLAP) and weigh their latency, consistency, and cost.Anda merancang arsitektur hot dan cold beserta serving layer (Lambda vs Kappa, real-time OLAP) dan menimbang latency, konsistensi, serta biayanya.
- 📄 Slide Pertemuan 12 v07112026
Data-driven: from metrics to dashboards and data appsData-driven: dari metrik ke dashboard dan data app
You present data for business decisions through metrics, a semantic layer, dashboards (Superset, Metabase), and data apps.Anda menyajikan data untuk keputusan bisnis melalui metrik, semantic layer, dashboard (Superset, Metabase), dan data app.
- 📄 Slide Pertemuan 13 v13112026
Data securityData security
You secure a data platform from ingestion to serving: encryption in transit and at rest, secret management, access control, and personal data protection.Anda mengamankan platform data dari ingestion sampai serving: enkripsi in transit dan at rest, secret management, kontrol akses, dan perlindungan data pribadi.
- 📄 Slide Pertemuan 14 v13112026
Data governance, GRC, and observabilityData governance, GRC, dan observability
You apply data governance (catalog, lineage, data contracts), GRC including Indonesia's Personal Data Protection Law, and observability to a data platform, and present your major project progress.Anda menerapkan data governance (catalog, lineage, data contract), GRC termasuk UU PDP, dan observability pada platform data, serta mempresentasikan progres tugas besar.
- 📄 Slide Pertemuan 15 v13112026
Final - group project: a large-scale Spark pipeline through to a dashboard or appUAS - tugas besar: pipeline data skala besar dengan Spark sampai dashboard atau aplikasi
Final exam as a major project, "Large-scale data pipeline with Apache Spark through to a dashboard or app": a bronze-silver-gold pipeline on a Spark cluster, proof of scaling, serving to a dashboard or data app, plus security and governance controls.Ujian Akhir Semester berupa tugas besar "Pipeline data skala besar dengan Apache Spark sampai dashboard atau aplikasi": pipeline bronze-silver-gold di klaster Spark, bukti scaling, serving ke dashboard atau data app, plus kontrol security dan governance.
- 📄 Brief Tugas Besar UAS v13112026