Playbook · v1.0
Design, build, and operate data systems
with one consistent shape.
A practical toolkit for engineers and analysts. Patterns, schemas, runbooks, and copy-paste templates — from a single laptop to Snowflake.
One mental model
Bronze → Silver → Gold across every deployment.
Four deployments
Local DuckDB, AWS, Railway, Snowflake.
Strict naming
Predictable schemas, tables, and jobs.
Audit by default
Pipeline runs and validation rules baked in.
All sections
27 chapters- 01OverviewWhat this playbook is and how to use it.
- 02How to Read ThisHow to use this site whether you're technical or not.
- 03Why This MattersWhat structured data systems prevent and enable.
- 04Design PhilosophyFive principles behind every system in this playbook.
- 05Start HereWho this playbook is for and the implementation flow.
- 06Decision FrameworkPick the right system and know when to migrate.
- 07End-to-End ExampleDriver Compliance System — CSV to Excel.
- 08Same Shape, Different ScaleBronze/Silver/Gold across DuckDB, Postgres, Snowflake.
- 09Operations StackRecommended orchestration, testing, CI/CD, and alerting.
- 10Core PrinciplesEngineering principles that anchor every system.
- 11System Design FlowSource → Bronze → Silver → Gold → Outputs.
- 12Build Your First Data SystemSeven concrete steps from CSV to Excel output.
- 13When to Use Each SystemPick the right deployment for your scenario.
- 14Local Data System (DuckDB)Single-laptop analytics stack with DuckDB + Python.
- 15Cloud Data System (Postgres / AWS)S3 → transform → Postgres → BI/API.
- 16Railway Low-Cost SystemTiny-team stack on Railway: Postgres + workers + cron.
- 17Snowflake / Enterprise SystemWarehouse-first architecture with governance.
- 18Naming ConventionsStrict patterns for schemas, tables, and jobs.
- 19Schema TemplatesBronze / Silver / Gold SQL you can paste in.
- 20Validation & Data QualityRules, logging tables, run tracking.
- 21How to Run the SystemExecution commands in order.
- 22Orchestration & SchedulingCron and Airflow/Dagster patterns.
- 23RunbooksRun, debug, backfill, recover.
- 24Architecture DiagramsReference diagrams for each deployment.
- 25Copy-Paste TemplatesFolder structures, SQL, Python skeletons.
- 26Project Kickoff ChecklistChecklist to ensure consistency before starting any data system.
- 27New Project TemplateFolder structure, requirements, and main.py base template.