7CS516 — Processing Big Data
Postgraduate module on storing, querying and processing data at scale, from SQL and data warehouses to NoSQL, MapReduce, Spark and streaming.
Instructor: Ioannis Tsioulis
Term: Level 7 (PG)
Location: Mediterranean College, Athens
Overview
This postgraduate module follows data as it outgrows the tools built for small amounts of it. Students start from the relational model and SQL, move to data warehouses and to storage formats and NoSQL systems designed for volume and variety, and then to computation that no longer fits on one machine: MapReduce, Apache Spark and stream processing. Throughout, the emphasis is on understanding what each technology does, measuring what it costs, and judging when it is the right tool and when a simpler one will do.
Topics
- The relational model, keys, SQL and transactions
- Joins, aggregation and window functions
- Indexes and query plans
- Data warehouses: facts, dimensions and the star schema
- Storage formats: row and column storage, CSV and Parquet
- NoSQL: document and key-value stores, sharding, replication and the CAP theorem
- Processing data that is larger than memory
- MapReduce: divide, map, shuffle, reduce
- Apache Spark: RDDs, DataFrames, Spark SQL and pipelines
- Stream processing: windows and late data
- The cloud, its costs, and end-to-end data pipelines
- Statistical analysis and data visualisation
- Machine learning: regression, classification and clustering, also at scale with Spark MLlib
- Artificial neural networks and deep learning
Companion notebooks
The open course From One Table to a Cluster: Working with Data at Scale was written to support this module. Its 18 hands-on notebooks run in Google Colab, with nothing to install, and build each idea by hand before handing it to a professional tool. Its last module, notebooks 13 to 18, takes a real data warehouse through the whole course and accompanies the module’s lab sessions. The table shows where each topic of the module is covered.
| Topic | Notebooks in From One Table to a Cluster |
|---|---|
| The relational model and SQL | 1, 2 |
| Indexes and query performance | 3 |
| Data warehouses | 4 |
| Storage formats | 5 |
| NoSQL systems | 6 |
| Data larger than memory | 7 |
| MapReduce | 8 |
| Apache Spark | 9, 10 |
| Stream processing | 11 |
| The cloud and end-to-end pipelines | 12 |
| A real data warehouse, end to end: profiling, building, documents, exploration with Spark, Spark MLlib, measuring scale | 13 to 18 |
The Python, pandas and visualisation needed for the practical work are covered in From Python to Deep Learning and Fine-Tuning, notebooks 1 to 7, and the machine learning and neural network topics in notebooks 8 to 13 of the same course.