Repository navigation
Showcase improvements - #1
Merged
Merged
Conversation
- Add .gitignore rules for *.duckdb, .DS_Store, dbt target/logs, .user.yml, .env, and Airflow runtime dirs - Untrack committed build artifacts and OS cruft: receipts.duckdb (pipeline output), .DS_Store files, logs/dag_processor/latest, dbt .user.yml, empty .env - Remove dead code dags/constants.py (unused CREATE TABLE DDL, superseded by dbt models; contained a 'cusomter_points' typo) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Rename project airbnb -> receipts_analytics (dir, dbt_project.yml, profile)
- Layered structure with materialization strategy:
staging (views) stg_* — thin renames, date parsed to DATE once
intermediate (ephemeral) int_* — business logic as inlined CTEs
marts (tables) fct_/dim_* — queried by the dashboard
- Convert pandas Python models to native DuckDB SQL (list unnesting),
removing pandas from the transform layer
- Fix models that referenced tables without ref()/source() (sale_profits,
sales_details) so dbt lineage is correct and complete
- Add source definition + 44 data tests (not_null, unique, relationships,
accepted_values, dbt_utils.unique_combination_of_columns) and real
column/model descriptions across every layer
- Add fct_employee_turnover: computes turnover per business per fiscal year
from receipt activity and flags the right-censored final year (replaces the
hand-counted dictionary that lived in the Streamlit page)
- Drop dead models (q3/products, q3/sales_details), unused drop_table macro,
and the leftover jaffle_shop README
- Add dbt_utils dependency; switch profile off unused httpfs extension
Verified: dbt build --target dev => PASS=56 (12 models, 44 tests), 0 errors
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Replace the single 'dbt run' BashOperator with a Cosmos DbtDag that renders the project into one task per model + its tests (TestBehavior.AFTER_EACH), with default_args (retries, retry_delay) and dbt run in the project .venv - Rewrite the ingestion asset: land main.raw_receipts via DuckDB read_json (typed structs/lists, no pandas), resolve paths from repo root (no CWD dependency), log row counts, and fail loudly on an empty load - Asset-schedule the dbt DAG on the raw_receipts ingestion asset - Pin base image apache/airflow:latest -> 3.0.6 for reproducible builds - Stop baking the DuckDB into the Streamlit image; mount the live pipeline output (read-only) so the dashboard reflects the latest dbt build Verified: dbt ls lists 18 models; Cosmos renders the DbtDag into 30 tasks (12 model runs + 12 tests); ingestion asset imports under Airflow 3.0.6 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Fix Q2 bug where both charts plotted num_purchases (the ternary compared value_type to the literal 'value_type'); 'amount spent' now shows spend - Collapse the four duplicated per-business subplot blocks into one loop - Rewrite Q3 to read fct_employee_turnover / dim_employee_attendance instead of a hand-counted Python dictionary; add a business filter, a turnover trend chart, and an explicit note on the censored final year - Centralise data access in constants.run_query: cached, read-only DuckDB connection (won't contend with the pipeline's write lock) with a friendly error if the database hasn't been built yet - Point all queries at the new model names (raw_receipts / stg_/ int_/ fct_) and resolve the DuckDB path from the repo root (CWD-independent) - Materialize intermediate models as views so the dashboard can inspect them - Pin dashboard requirements (streamlit/duckdb/plotly/pandas) Verified: streamlit AppTest runs all three pages with no exceptions against the built DuckDB Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add .github/workflows/ci.yml: on every push/PR, seed the landing table, run 'dbt build' (18 models + 44 tests), then the dashboard smoke test - Add scripts/build_local_db.py (seed raw_receipts from JSON without Airflow) and scripts/smoke_test_dashboard.py (headless AppTest of all pages) - Rewrite README as a portfolio piece: architecture diagram, dbt layer/model table, run instructions (Docker + local dbt), testing/CI, design decisions, and results framed as outcomes (incl. the turnover censoring explanation) - Rewrite .env.sample to document the real docker-compose variables and drop the unused MotherDuck/RapidAPI leftovers Verified locally: full CI sequence (seed -> dbt build PASS=62 -> smoke test) passes in a single environment Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dbt-core is provided transitively by dbt-duckdb; the standalone 'dbt' package (pinned at 1.0.0.40.5) is deprecated and installed nothing useful. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
python:3.12-slim floated to trixie, where apt can't find software-properties-common (and build-essential was unnecessary). Pin to slim-bookworm and install only curl (needed for the compose healthcheck); all dashboard deps install from wheels. Verified: docker build -f Dockerfile_streamlit succeeds. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The images built but the runtime was broken. Fixes: - Cosmos: render from a pre-built manifest (LoadMode.DBT_MANIFEST) instead of shelling out to 'dbt ls' on every parse, which exceeded the 30s DAG import timeout. Manifest is compiled at image build time (dbt deps && dbt parse). - Use the image's dbt (/home/airflow/.local/bin/dbt) — the previously copied host .venv is non-relocatable (its shebang points at the host path) and was broken in the container. Drop the COPY .venv and its compose mount. - Reference the raw_receipts asset via Asset(...) instead of importing it into dbt.py, which had re-registered the DAG (AirflowDagDuplicatedIdException). - Separate code from data: bake the dbt project (manifest + dbt_packages) into the image; share only the DuckDB via a named 'warehouse' volume mounted by both Airflow (writer) and Streamlit (reader). Update profiles prod path, the ingestion path, and the dashboard path (env-driven) accordingly. - COPY --chown=airflow:0 dbt/ so dbt can write packages/manifest at build. - Slim requirements to the packages actually used (drop unused astro-sdk-python, airflow-provider-duckdb, minio that also caused resolver conflicts). Verified in the built image: both DAGs import with no errors in ~3s (dbt_receipts_analytics = 30 tasks, asset-triggered), and 'dbt build --target prod' completes successfully against the warehouse volume. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cosmos renders one Airflow task per model/test; under LocalExecutor these ran in parallel and collided on DuckDB's single write lock (tasks failed and retried, stalling the run). Set max_active_tasks=1 / max_active_runs=1 so the dbt DAG builds one model at a time. Also lower retry_delay 5m -> 1m (the whole local pipeline runs in ~2-3 min). Verified end-to-end via docker compose up: raw_receipts ingestion asset -> asset-triggered dbt DAG -> 19 objects built (7 staging + 6 intermediate + 5 marts) -> DAG run state 'success'; Streamlit reads the marts live from the shared warehouse volume. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The smoke test segfaulted (exit 139) on the Linux x86_64 CI runner. Running several Streamlit AppTests in a single interpreter accumulates process-global runtime state and crashes the native duckdb/streamlit extensions. Each page passes in isolation, so run each in its own subprocess. Reproduced the segfault in a linux/amd64 container and verified the fix there (and locally): all three pages pass, exit 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.