Skip to content

README.md

Latest commit

 

History

History
342 lines (258 loc) · 25.4 KB

README.md

File metadata and controls

342 lines (258 loc) · 25.4 KB
Raw

Herald

One who reads and announces markings.

Herald is a Go web service with an embedded Lit web client for classifying approximately 1,000,000 Department of Defense PDF documents' security markings using Azure AI Foundry GPT models. It reads security classification markings from document pages, interprets them through vision-capable LLMs, and produces structured classification records suitable for database association and downstream system integration.

Vision

Organizations managing large corpora of classified documents face a persistent bottleneck: manually reading and recording security markings across millions of pages. Herald eliminates this bottleneck by applying vision-capable LLMs to interpret markings at scale, producing structured classification records that humans validate and downstream systems consume. The service is designed for rapid delivery, drawing heavily from proven patterns established during R&D in agent-lab and the tau agentic-units ecosystem.

Core Premise

Herald occupies a deliberate middle ground between two existing implementations:

  • classify-docs (go-agents/tools/classify-docs): A CLI tool using pure sequential page-by-page processing with context accumulation. Accurate (96.3%) but no web interface, no persistence, no batch processing.

  • agent-lab: A full workflow experimentation platform with 5-node classification graphs, parallel detection, observer infrastructure, profile management, and image caching. Powerful but far more infrastructure than Herald needs.

Herald takes the per-page vision analysis approach from classify-docs, wraps it in a 4-node state graph from go-agents-orchestration (init, classify, conditional enhance, finalize), and hosts it in a streamlined Go web service adapted from agent-lab's infrastructure patterns. Pages are classified in parallel with bounded concurrency, and a dedicated finalize node synthesizes the document-level classification from all page findings. Documents flow in through the service API to Azure Blob Storage, classification results flow out through PostgreSQL, and humans validate results through an embedded Lit web client.

The architecture deliberately excludes: image caching (images are ephemeral during classification), observer/checkpoint infrastructure (results self-contain context), multi-workflow registries (single classification workflow), provider/agent CRUD (single externally-configured agent), and OpenAPI/Scalar documentation (deferred for velocity).

Phases

Phase Focus Area Version Target Status
Phase 1 - Service Foundation Go project scaffolding, Azure PostgreSQL schema/migrations, Azure Blob Storage integration, configuration system, module/routing infrastructure, document domain (upload single + batch, registration, metadata), storage abstraction, lifecycle coordination v0.1.0 Complete
Phase 2 - Classification Engine Agent configuration from external config, classification workflow state graph (init -> classify -> enhance? -> finalize), parallel per-page analysis with bounded concurrency, named prompt overrides (DB + API), single document classification endpoint, classification result schema with flattened workflow metadata v0.2.0 Complete
Phase 3 - Web Client Lit 3.x SPA with native Bun builds embedded in Go binary, Air hot reload for development, document management UI (upload single + batch, browse, search, filter, bulk classify), SSE streaming observer for classification progress, classification result viewing/validation/manual adjustment with PDF viewer, prompt modification management v0.3.0 Complete
Phase 4 - Security and Deployment Azure Entra authentication (opt-in JWT middleware + MSAL.js web client), managed identity for Azure services (Storage, PostgreSQL, AI Foundry), Docker containerization with ImageMagick 7.0+, Azure Container Apps deployment configuration, IL4/IL6 environment configuration v0.4.0 Complete
Phase 5 - Post-Deployment Polish Post-IL6-deployment quality-of-life adjustments: document-upload scroll container, configurable page size selector, URL query-parameter state persistence across view transitions, migration from go-agents/go-agents-orchestration to tau/agent and tau/orchestrate, MSAL stale credential refresh hardening, native top-layer overlay primitives (dialog/popover), toast service for command feedback, format-aware document processing (PDF + raw PNG/JPEG/WEBP) with per-format handler registry replacing the document-context dependency v0.5.0 Complete

Architecture

Project Structure

herald/
├── cmd/
│   ├── server/           # HTTP server entry point and composition
│   └── migrate/          # Database migration CLI
├── internal/
│   ├── api/              # API module: Runtime, Domain, route registration
│   ├── config/           # Configuration management (JSON + env overlays)
│   ├── infrastructure/   # Infrastructure assembly (lifecycle, logger, database, storage, agent)
│   ├── state/            # Shared classification types (ClassificationState, ClassificationPage, EnhanceSettings, Confidence, state keys)
│   ├── format/           # Document-format registry + PDF and image handlers + shared magick helper
│   ├── documents/        # Document domain (upload, registration, metadata, blob management, content-type validation)
│   ├── classifications/  # Classification result domain (store, query, validate, adjust)
│   ├── prompts/          # Named prompt override domain (CRUD, per-stage loading)
│   └── workflow/          # Classification workflow definition
│       ├── workflow.go    # State graph assembly: init -> classify -> enhance? -> finalize
│       ├── init.go        # Init node: dispatch to format handler for extraction
│       ├── classify.go    # Classify node: parallel per-page analysis with bounded concurrency
│       ├── enhance.go     # Enhance node: dispatch to format handler for conditional re-render
│       ├── finalize.go    # Finalize node: document-level classification synthesis
│       ├── types.go       # WorkflowResult (execution metadata wrapping state.ClassificationState)
│       └── prompts.go     # Prompt composition with instructions, specs, and running state
├── pkg/
│   ├── auth/             # Auth config, User context helpers, JWT middleware (opt-in)
│   ├── core/             # Stateless cross-cutting primitives (byte formatting, JSON extraction, bounded concurrency)
│   ├── database/         # PostgreSQL connection management (pgx driver)
│   ├── handlers/         # HTTP response utilities (RespondJSON, RespondError)
│   ├── lifecycle/        # Startup/shutdown coordination
│   ├── middleware/       # HTTP middleware (CORS, logging)
│   ├── module/           # Modular HTTP routing (prefix-based modules)
│   ├── pagination/       # PageRequest/PageResult pattern
│   ├── query/            # SQL query builder with projections and sorting
│   ├── repository/       # Database helpers (QueryOne, QueryMany, WithTx)
│   ├── routes/           # Route registration types
│   ├── storage/          # Blob storage interface (Azure Blob Storage implementation)
│   └── web/              # Template set, router, embedded asset serving
├── app/                  # Lit 3.x SPA
│   ├── app.go            # Go module: embed dist/*, shell template, NewModule()
│   ├── package.json      # Bun project (no Vite)
│   ├── scripts/          # Bun build scripts (build.ts, watch.ts)
│   ├── client/           # TypeScript source (native Bun build)
│   ├── dist/             # Build output (app.js, app.css)
│   └── server/           # Go HTML templates (layouts/app.html)
├── deploy/               # Bicep IaC deployment manifests for Azure Container Apps
├── config.json           # Base configuration
├── .air.toml             # Air hot reload configuration
├── Dockerfile
└── _project/
    └── README.md         # This document

Layered Composition Architecture

Herald follows the Layered Composition Architecture (LCA) established in agent-lab:

Cold Start (initialization, no connections):

  1. config.Load() -- Read JSON base + environment overlay + env var overrides
  2. infrastructure.New(cfg) -- Create lifecycle coordinator, logger, database system, storage system, validate agent config
  3. NewModules(infra, cfg) -- Create API module (runtime + domain assembly) and web app module

Hot Start (connections established, ready to serve):

  1. infra.Start() -- Database connect + pool, storage container initialization
  2. http.Start(lifecycle) -- Begin listening, register shutdown hook
  3. lifecycle.WaitForStartup() -- All subsystems report ready

Shutdown (coordinated teardown):

  1. Signal received (SIGINT/SIGTERM)
  2. lifecycle.Shutdown(timeout) -- Reverse-order hook execution within deadline

Infrastructure Layer

type Infrastructure struct {
    Lifecycle  *lifecycle.Coordinator
    Logger     *slog.Logger
    Database   database.System
    Storage    storage.System                              // Azure Blob Storage implementation
    Agent      tauconfig.AgentConfig                       // tau protocol/config for per-request agent creation
    Credential azcore.TokenCredential                      // Azure managed identity (nil when auth disabled)
    NewAgent   func(ctx context.Context) (agent.Agent, error) // Auth-mode-aware agent factory
}

API Module Composition

type Runtime struct {
    *infrastructure.Infrastructure
    Pagination pagination.Config
}

type Domain struct {
    Documents       documents.System
    Classifications classifications.System
    Prompts         prompts.System
}

Domain Systems

Each domain follows the handler pattern: a System interface, repository with query builder, Handler struct with Routes() method.

documents -- Upload (single + batch), registration, metadata, blob lifecycle. Documents are immutable after upload.

classifications -- 1:1 relationship with documents. Stores the classification result with flattened workflow metadata columns. Supports manual validation and adjustment.

prompts -- Named prompt instruction overrides stored in PostgreSQL with CRUD API endpoints. Each override targets a specific workflow stage and provides tunable instructions. The final system prompt is composed of two layers: instructions (tunable, from DB or hard-coded defaults) + output format (hard-coded, never exposed). The workflow assembles both layers at execution time.

Classification Workflow

A 4-node state graph using tau/orchestrate:

init --> classify --> [needs enhancement?] --> enhance --> finalize --> exit
                              |
                              v (no enhancement needed)
                          finalize --> exit

init node: Dispatches to the format handler registered for the document's content type. PDF handlers render each page to PNG via ImageMagick (using magick's native source.pdf[N] selector syntax, bounded concurrency via errgroup); image handlers copy PNG sources verbatim and normalize JPEG/WEBP sources to PNG. Output images land in a request-scoped temp directory that the workflow owns (created with os.MkdirTemp, removed via deferred RemoveAll).

classify node: Parallel per-page analysis using bounded errgroup concurrency. Each page is sent to the vision-capable GPT model independently (no accumulated context between pages). The model populates per-page ClassificationPage data (markings found, rationale, enhancement flags) but does not produce document-level classification — that is deferred to the finalize node. Pages flagged with Enhance: true include an Enhancements description of what adjustments are needed.

enhance node (conditional): Triggered when any page's Enhance flag is true (evaluated via ClassificationState.NeedsEnhance()). Re-renders flagged pages in parallel with bounded errgroup concurrency by delegating to the same format handler used during init, applying EnhanceSettings filters (brightness, contrast, saturation) and reclassifying via vision. Trigger conditions TBD through experimentation during Phase 2; initially, classify never sets Enhance: true.

finalize node: Always runs as the terminal node. Performs a single inference that reviews all per-page analysis results (including any enhanced pages) and produces the authoritative document-level ClassificationState fields: classification, confidence, and rationale. Sees all evidence holistically rather than incrementally.

This collapses agent-lab's 5-node graph (init, detect, enhance, classify, score) into 4 nodes by merging detection into the classify node, separating document-level synthesis into a dedicated finalize node, and keeping enhance as optional remediation. The classify node runs one LLM call per page; finalize runs one LLM call total.

Agent Configuration

Single agent definition from external configuration (not CRUD-managed). Uses tau's protocol/config.AgentConfig type directly — no Herald-specific agent config structs:

{
  "agent": {
    "name": "herald-classifier",
    "provider": {
      "name": "azure",
      "base_url": "https://...",
      "options": {
        "deployment": "gpt-5-mini",
        "api_version": "2024-12-01-preview",
        "auth_type": "api_key"
      }
    },
    "model": {
      "name": "gpt-5-mini",
      "capabilities": {
        "vision": {
          "max_tokens": 4096,
          "temperature": 0.1
        }
      }
    }
  }
}

Configured at startup from JSON config + env var overrides. Token injected via HERALD_AGENT_TOKEN env var. Provider, model, and provider options are also overridable via HERALD_AGENT_* env vars.

Storage Architecture

Azure Blob Storage with flat layout:

documents/{document-id}/{filename}

Documents are immutable after upload. Processing status is tracked exclusively in the PostgreSQL database. All documents enter through the Go service API (single or batch upload) to ensure database record + blob atomicity.

Database Schema

Azure PostgreSQL with golang-migrate. Core tables:

documents -- Registration records with metadata and blob reference. Status tracks a document's position in the classification lifecycle, not operational concerns. pending means inference hasn't completed; review means inference produced a result awaiting human validation; complete means the classification has been validated or adjusted. Errors during inference leave the document in pending — error handling and observability are separate concerns.

  • id, external_id, external_platform, filename, content_type, size_bytes, page_count, storage_key, status (pending/review/complete), uploaded_at, updated_at

classifications -- 1:1 with documents, overwritten on re-classification. Workflow metadata is stored as flattened columns rather than JSONB. Human review is either validate (agree with AI) or update (overwrite classification and rationale). Both transition the document to complete.

  • id, document_id (unique FK), classification, confidence, markings_found (JSONB), rationale, classified_at, model_name, provider_name
  • validated_by, validated_at

prompts -- Named instruction overrides per workflow stage. Stores the tunable instruction layer; the output format layer is hard-coded in workflow/prompts.go and composed at execution time.

  • id, name (unique), stage (classify/enhance/finalize), instructions, description, active

Web Client

Lit 3.x SPA following agent-lab patterns:

  • Build: Native Bun builds with TypeScript (Bun.build() API, no Vite). Component CSS via { type: 'css' } import attribute — Bun plugin emits CSSStyleSheet (Lit 3+ accepts directly). Global CSS extracted automatically.
  • Development: Air hot reload for Go + Bun watch for TypeScript/CSS. Two-terminal workflow: bun run watch rebuilds assets, air rebuilds and restarts the Go server on dist/ changes.
  • Embedding: go:embed dist/*, shell template pattern
  • Routing: Client-side History API router, component prefix hd-
  • Styling: CSS cascade layers with design tokens
  • State: Lit @state() decorators for component-local reactivity
  • Services: Stateless PascalCase service objects with base path constants

Key views: document management (upload, browse, classify, bulk operations), prompt management (CRUD), document review (PDF viewer alongside classification for validation/adjustment).

Key Decisions

Decision Choice Rationale
Workflow topology 4-node (init -> classify -> enhance? -> finalize) Classify handles per-page analysis (one LLM call per page), finalize synthesizes document-level classification from all page findings (one LLM call total). Separating per-page analysis from document-level synthesis eliminates incremental anchoring bias and ensures finalize sees all evidence (including enhanced pages) holistically.
Classification approach Parallel per-page analysis, finalize synthesis Classify processes all pages concurrently with bounded errgroup concurrency (no accumulated context). Finalize produces the authoritative document classification from complete page data.
Image lifecycle Ephemeral (render, encode, discard) 1M documents would generate enormous image storage. Images serve only as Vision API input.
Classification result model 1:1 with document, overwritten on re-classification Simpler than run/stage/decision tracking. Flattened workflow metadata columns preserve provenance.
Agent configuration Single externally-configured agent Herald serves one purpose with one or two GPT models. External config matches deployment patterns.
Prompt management Two-layer composition: tunable instructions (DB overrides or hard-coded defaults) + hard-coded output format Lighter than agent-lab's profile + profile_stages system. Instructions are tunable without risking broken output formats.
Blob storage layout Flat with DB-driven status Avoids expensive blob-move operations across 1M documents. DB is single source of truth.
Database Azure PostgreSQL Maximizes code reuse from agent-lab (pgx, query builder, repository patterns).
Observability SSE streaming observer (Phase 3) Classification progress uses Server-Sent Events — unidirectional server-push, works behind standard load balancers, auto-reconnects natively. tau/orchestrate supports observer injection (cfg.Observer); implementation deferred to Phase 3 alongside the web client that drives its design. WebSockets rejected — upgrade negotiation, sticky sessions, and message broker requirements for cluster deployments are unjustified for unidirectional progress streaming.
Document format handling Per-format handler registry (PDF, raw image) A single format.Handler interface with content-type dispatch decouples init/enhance/upload-validation from any one format's rasterizer. PDF uses pdfcpu (page count) + magick (native source.pdf[N] selector). Image normalizes JPEG/WEBP to PNG. New formats (DOCX, PPTX, TIFF) land as additional handlers without threading branches through the workflow or client.
Batch classification Client-orchestrated parallel single-document classifications Same pattern as document uploads — deterministic per-document behavior. Clients coordinate via Promise.allSettled.
Bulk upload Sequential single-file uploads (no batch endpoint) ParseMultipartForm(maxMemory) caps total request memory, making a batch endpoint's per-file size limit unpredictable. Single-file uploads give deterministic per-file size limits. The web client coordinates multi-file uploads via <input multiple> with Promise.allSettled, providing per-file progress, retry, and error handling.
Web client scope Full management MVP Upload, browse, classify, validate, monitor, manage prompts. Complete operational interface.

Dependencies

Go Libraries (ecosystem)

  • tau/agent: LLM abstraction. Agent interface, Vision/Chat/Tools/Embed protocol methods, typed request/response. Pulls in sibling modules tau/protocol (messages, config), tau/format (wire format registry, openai format), tau/provider (provider registry, azure/ollama factories).
  • tau/orchestrate: State graph workflow engine. StateGraph, StateNode, conditional edges, observability events.

Go Libraries (external)

  • pgx: PostgreSQL driver with connection pooling
  • golang-migrate: Database migration management
  • google/uuid: UUID generation
  • azure-sdk-for-go: Azure Blob Storage client, Azure Identity (Entra auth)
  • pdfcpu: PDF page count extraction (upload + format/pdf handler)
  • go-oidc: OIDC discovery, JWKS management, and JWT token verification (coreos/go-oidc)

Dependency Criteria

External dependencies are acceptable when they meet all of the following:

  • Purpose-built — solves a specific, well-scoped problem (not a kitchen-sink framework)
  • Authoritative source — maintained by a reputable organization or established in the ecosystem
  • Actively maintained — recent releases, responsive to issues, not abandoned
  • Production-ready — stable API, used in production by major projects
  • Not a supply chain risk — minimal transitive dependencies, auditable codebase

Frontend

  • Lit 3.x (lit)
  • Bun: JavaScript runtime, package manager, and bundler (build-time)
  • Air: Go live reload for development (github.com/air-verse/air)

Runtime

  • ImageMagick 7.0+ with Ghostscript PDF delegate: PDF rasterization and raw-image (JPEG/WEBP) normalization to PNG (required in deployment containers; apk add imagemagick ghostscript in Dockerfile)
  • Azure PostgreSQL: Managed database service
  • Azure Blob Storage: Document persistence
  • Azure AI Foundry: GPT-5-mini and GPT-5.2 deployments (both confirmed available on IL6)

Integration Points

Azure AI Foundry

  • Single configurable agent targeting GPT-5-mini or GPT-5.2 per deployment
  • Vision API for page classification (base64 data URI encoded images)
  • Authentication via API key (Azure Key Vault) or access token
  • Configuration: tau protocol/config.AgentConfig in config.json + HERALD_AGENT_* env var overrides

Azure Blob Storage

  • Document upload via Go service API (ensures DB + blob atomicity)
  • Flat key structure: documents/{document-id}/{filename}.pdf
  • Azure SDK azblob client with managed identity or connection string auth
  • Immutable blobs (upload once, delete on removal, never modify)

Azure PostgreSQL

  • pgx driver with connection pooling
  • golang-migrate for schema management
  • Query builder pattern for dynamic filtering and sorting

Azure Entra ID

  • Service authentication: user-assigned managed identity for Blob Storage, PostgreSQL, AI Foundry access
  • API authentication: opt-in JWT bearer middleware via go-oidc OIDC discovery and token verification
  • Web client authentication: MSAL.js with redirect login flow, token injection into API requests
  • User identity: JWT claims stored in validated_by for audit trail (no OBO — managed identity for service-to-service)
  • Token management: Azure Identity SDK credential chain, pgx BeforeConnect hook for database token refresh

External Document Sources

  • Documents enter Herald exclusively through the upload API (single or batch)
  • External systems push documents to Herald; Herald does not pull
  • Document metadata fields (external_id, external_platform) for external system linkage

Open Questions

  1. Enhance stage trigger conditions: What quality thresholds trigger conditional enhancement? The classify node must report whether image quality was a limiting factor.
  2. GPT-5-mini vs GPT-5.2 benchmarking: Which model performs better at acceptable cost? Both confirmed available on IL6.
  3. Bulk ingestion strategy: Loading 1M documents requires a bulk ingestion approach feeding the upload API. Detailed strategy TBD.

Resolved Questions

  1. External system identifier fields: external_id and external_platform columns on the documents table provide linkage back to originating systems. Resolved in Phase 1.
  2. Confidence scoring approach: Categorical (HIGH/MEDIUM/LOW). Aligns with classify-docs' proven approach and is more interpretable for human validators.
  3. Worker pool sizing: Not applicable — no batch classification endpoint. Clients orchestrate parallel single-document classifications.
  4. SSE vs WebSockets for classification progress: SSE. Classification progress is unidirectional (server → client). SSE works over HTTP/1.1, requires no upgrade negotiation, operates behind standard load balancers and reverse proxies, and auto-reconnects natively. WebSockets would be justified only if the client needed bidirectional communication during classification. Cancel can be a separate REST endpoint. go-agents-orchestration already supports observer injection — the implementation point is workflow.go's cfg.Observer = "noop", swappable to a StreamingObserver ported from agent-lab. Full SSE implementation path documented in the Phase 2 review plan.
  5. Azure deployment target: Azure Container Apps. Single-service deployment, no cluster management needed. Bicep IaC with modular templates. Resolved in Phase 4.
  6. Remove PageRequest from core: Each domain defines its own SearchRequest with domain-specific filters. PageRequest removed from @core, toQueryString retained as a generic utility. Resolved in Phase 4.