Skip to content

Showcase improvements - #1

Merged
jackptoke merged 12 commits into
mainfrom
showcase-improvements
Jul 10, 2026
Merged

jackptoke merged 12 commits into
mainfrom
showcase-improvements

Conversation

@jackptoke

Copy link
Copy Markdown
Owner

No description provided.

karieng-com-au and others added 12 commits July 10, 2026 11:56
- Add .gitignore rules for *.duckdb, .DS_Store, dbt target/logs,
  .user.yml, .env, and Airflow runtime dirs
- Untrack committed build artifacts and OS cruft:
  receipts.duckdb (pipeline output), .DS_Store files,
  logs/dag_processor/latest, dbt .user.yml, empty .env
- Remove dead code dags/constants.py (unused CREATE TABLE DDL,
  superseded by dbt models; contained a 'cusomter_points' typo)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Rename project airbnb -> receipts_analytics (dir, dbt_project.yml, profile)
- Layered structure with materialization strategy:
    staging      (views)     stg_* — thin renames, date parsed to DATE once
    intermediate (ephemeral) int_* — business logic as inlined CTEs
    marts        (tables)    fct_/dim_* — queried by the dashboard
- Convert pandas Python models to native DuckDB SQL (list unnesting),
  removing pandas from the transform layer
- Fix models that referenced tables without ref()/source() (sale_profits,
  sales_details) so dbt lineage is correct and complete
- Add source definition + 44 data tests (not_null, unique, relationships,
  accepted_values, dbt_utils.unique_combination_of_columns) and real
  column/model descriptions across every layer
- Add fct_employee_turnover: computes turnover per business per fiscal year
  from receipt activity and flags the right-censored final year (replaces the
  hand-counted dictionary that lived in the Streamlit page)
- Drop dead models (q3/products, q3/sales_details), unused drop_table macro,
  and the leftover jaffle_shop README
- Add dbt_utils dependency; switch profile off unused httpfs extension

Verified: dbt build --target dev => PASS=56 (12 models, 44 tests), 0 errors

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Replace the single 'dbt run' BashOperator with a Cosmos DbtDag that renders
  the project into one task per model + its tests (TestBehavior.AFTER_EACH),
  with default_args (retries, retry_delay) and dbt run in the project .venv
- Rewrite the ingestion asset: land main.raw_receipts via DuckDB read_json
  (typed structs/lists, no pandas), resolve paths from repo root (no CWD
  dependency), log row counts, and fail loudly on an empty load
- Asset-schedule the dbt DAG on the raw_receipts ingestion asset
- Pin base image apache/airflow:latest -> 3.0.6 for reproducible builds
- Stop baking the DuckDB into the Streamlit image; mount the live pipeline
  output (read-only) so the dashboard reflects the latest dbt build

Verified: dbt ls lists 18 models; Cosmos renders the DbtDag into 30 tasks
(12 model runs + 12 tests); ingestion asset imports under Airflow 3.0.6

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Fix Q2 bug where both charts plotted num_purchases (the ternary compared
  value_type to the literal 'value_type'); 'amount spent' now shows spend
- Collapse the four duplicated per-business subplot blocks into one loop
- Rewrite Q3 to read fct_employee_turnover / dim_employee_attendance instead
  of a hand-counted Python dictionary; add a business filter, a turnover trend
  chart, and an explicit note on the censored final year
- Centralise data access in constants.run_query: cached, read-only DuckDB
  connection (won't contend with the pipeline's write lock) with a friendly
  error if the database hasn't been built yet
- Point all queries at the new model names (raw_receipts / stg_/ int_/ fct_)
  and resolve the DuckDB path from the repo root (CWD-independent)
- Materialize intermediate models as views so the dashboard can inspect them
- Pin dashboard requirements (streamlit/duckdb/plotly/pandas)

Verified: streamlit AppTest runs all three pages with no exceptions against
the built DuckDB

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Add .github/workflows/ci.yml: on every push/PR, seed the landing table,
  run 'dbt build' (18 models + 44 tests), then the dashboard smoke test
- Add scripts/build_local_db.py (seed raw_receipts from JSON without Airflow)
  and scripts/smoke_test_dashboard.py (headless AppTest of all pages)
- Rewrite README as a portfolio piece: architecture diagram, dbt layer/model
  table, run instructions (Docker + local dbt), testing/CI, design decisions,
  and results framed as outcomes (incl. the turnover censoring explanation)
- Rewrite .env.sample to document the real docker-compose variables and drop
  the unused MotherDuck/RapidAPI leftovers

Verified locally: full CI sequence (seed -> dbt build PASS=62 -> smoke test)
passes in a single environment

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dbt-core is provided transitively by dbt-duckdb; the standalone 'dbt'
package (pinned at 1.0.0.40.5) is deprecated and installed nothing useful.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
python:3.12-slim floated to trixie, where apt can't find
software-properties-common (and build-essential was unnecessary). Pin to
slim-bookworm and install only curl (needed for the compose healthcheck);
all dashboard deps install from wheels.

Verified: docker build -f Dockerfile_streamlit succeeds.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The images built but the runtime was broken. Fixes:
- Cosmos: render from a pre-built manifest (LoadMode.DBT_MANIFEST) instead of
  shelling out to 'dbt ls' on every parse, which exceeded the 30s DAG import
  timeout. Manifest is compiled at image build time (dbt deps && dbt parse).
- Use the image's dbt (/home/airflow/.local/bin/dbt) — the previously copied
  host .venv is non-relocatable (its shebang points at the host path) and was
  broken in the container. Drop the COPY .venv and its compose mount.
- Reference the raw_receipts asset via Asset(...) instead of importing it into
  dbt.py, which had re-registered the DAG (AirflowDagDuplicatedIdException).
- Separate code from data: bake the dbt project (manifest + dbt_packages) into
  the image; share only the DuckDB via a named 'warehouse' volume mounted by
  both Airflow (writer) and Streamlit (reader). Update profiles prod path, the
  ingestion path, and the dashboard path (env-driven) accordingly.
- COPY --chown=airflow:0 dbt/ so dbt can write packages/manifest at build.
- Slim requirements to the packages actually used (drop unused astro-sdk-python,
  airflow-provider-duckdb, minio that also caused resolver conflicts).

Verified in the built image: both DAGs import with no errors in ~3s
(dbt_receipts_analytics = 30 tasks, asset-triggered), and
'dbt build --target prod' completes successfully against the warehouse volume.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cosmos renders one Airflow task per model/test; under LocalExecutor these ran
in parallel and collided on DuckDB's single write lock (tasks failed and
retried, stalling the run). Set max_active_tasks=1 / max_active_runs=1 so the
dbt DAG builds one model at a time. Also lower retry_delay 5m -> 1m (the whole
local pipeline runs in ~2-3 min).

Verified end-to-end via docker compose up: raw_receipts ingestion asset ->
asset-triggered dbt DAG -> 19 objects built (7 staging + 6 intermediate +
5 marts) -> DAG run state 'success'; Streamlit reads the marts live from the
shared warehouse volume.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The smoke test segfaulted (exit 139) on the Linux x86_64 CI runner. Running
several Streamlit AppTests in a single interpreter accumulates process-global
runtime state and crashes the native duckdb/streamlit extensions. Each page
passes in isolation, so run each in its own subprocess.

Reproduced the segfault in a linux/amd64 container and verified the fix there
(and locally): all three pages pass, exit 0.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@jackptoke
jackptoke merged commit 0b67609 into main Jul 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants