Auditing

[hilmarf]

Project Proposal: Audit Logging SIG #2409

Overview

This pull request introduces a new Audit Logging signal to OpenTelemetry—a purpose-built observability pipeline for security-relevant events that satisfies compliance requirements such as ISO 27001, SOC 2, PCI-DSS, and HIPAA.

Motivation

The existing OpenTelemetry Logs signal is designed for general-purpose observability and intentionally permits sampling, back-pressure shedding, and record transformation. These behaviors are incompatible with audit logging, where:

• Every record MUST be delivered without loss or modification to the designated audit sink
• Records MUST NOT be sampled or dropped for any reason
• Integrity verification is mandatory for tamper-evidence

A dedicated Audit Logging signal provides:

• A purpose-built SDK pipeline with no sampling and at-least-once delivery semantics
• A distinct OTLP endpoint (/v1/audit) for network-level isolation from observability backends
• A dedicated data model with mandatory integrity fields and optional digital signatures
• Clear separation of audit records from operational logs

Changes

This PR adds comprehensive specification documentation and establishes the complete audit logging framework:

New Documents

• OTEP 0267 – Complete Audit Logging Signal proposal (motivations, design rationale, trade-offs)
• README.md – Introduction and signal overview
• api.md – Audit Logging API specification (AuditProvider, AuditLogger, emit semantics)
• data-model.md – AuditRecord and AuditReceipt data models with detailed field specifications
• sdk.md – SDK implementation requirements (queuing, processors, exporters, failure handling)
• collector.md – Tier-2 Collector specification for enterprise multi-sink deployments

Key Features

✅ Guaranteed Delivery – At-least-once delivery with durable (disk-backed) queuing
✅ Integrity Protection – Optional asymmetric signatures or symmetric HMACs for tamper-evidence
✅ No Partial Success – OTLP receivers reject entire batches if any record fails
✅ Clock Skew Detection – Dual timestamps (event vs. observed) enable clock synchronization verification
✅ Idempotency – RecordId as idempotency key prevents duplicate audit entries on retries
✅ Hash-Chain Support – Stream-scoped hash chains (audit.sequence.stream_id, audit.sequence.number, audit.sequence.previous_hash, audit.sequence.previous_record_id) for ordered, tamper-evident streams across multi-tenant deployments
✅ Graceful Stream End – audit.sequence.end signals a cleanly closed stream (distinguishes intentional termination from crashes)
✅ Compliance Ready – ISO 27001 Annex A requirements built into the core design

Data Model Highlights

AuditRecord – Mandatory fields:

• audit.record.id, Timestamp, ObservedTimestamp, EventName
• audit.actor.id, audit.actor.type, audit.action, audit.outcome
• Optional: audit.target.id/type, audit.source.id/type, audit.integrity.value, audit.integrity.signer, audit.integrity.canonicalization, audit.sequence.number, audit.sequence.previous_hash, audit.sequence.previous_record_id, audit.sequence.stream_id, audit.sequence.end, audit.schema.version

AuditReceipt – Proof-of-delivery returned by sink:

• RecordId (echoed), IntegrityHash (SHA-256), SinkTimestamp

Specification Status

• Status: Development
• Prototype Implementations: Planned for Java and Go SDKs
• OTLP Transport: Uses standard LogRecord protobuf with audit=true flag; dedicated /v1/audit endpoint

References

• Implements compliance requirements from ISO 27001 Annex A, SOC 2, PCI-DSS, HIPAA
• Extends OTEP 0092 (Logs Vision) and OTEP 0202 (Events and Logs API) patterns
• Reuses OTLP LogRecord as transport container for efficiency
• opentelemetry-collector-contrib/signingprocessor

[CLAassistant]

All committers have signed the CLA.