Sulba
ReviewSign inAccount
Theme
Editor theme
Light
Dark

Intermediate · Data pipelines

Testing a pipeline

Join a month of New York's cab trips to the taxi zones and to Central Park's weather with Spark, and test the pipeline that does it: NOAA's daily weather file read into units, each borough's trips on every day of August, wet days against dry days of the same weekday, a review of an assistant's daily table, and tests of your own that catch the mistakes planted in the pipeline.

You build
weather.py, then daily.py, then rain.py, then daily_table.py, then test_pipeline.py
Stages
3 Build, 1 Catch, 1 Ship
Time
About 3 hours
Runs on
Your machine

Loading the workspace…