Sulba
ReviewSign inAccount
Theme
Editor theme
Light
Dark

Intermediate · Data pipelines

Spark DataFrames

Work through a month of New York's yellow and green cab trips with Spark's DataFrame API: one table from two files laid out differently, a flag for each record that can't be taken at its word, each zone's days with a seven-day mean over the calendar, a review of an assistant's report, and daily Parquet files that a run replaces whole.

You build
trips.py, then flags.py, then days.py, then report.py, then store.py
Stages
3 Build, 1 Catch, 1 Ship
Time
About 3 hours
Runs on
Your machine

Loading the workspace…