Skip to content
Draft
Show file tree
Hide file tree
Changes from 65 commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
25b7c8e
🛡️ Sentinel: [HIGH] Fix DoS from hmac.compare_digest Unicode exception
seonghobae Aug 4, 2026
ac9b335
merge(develop): refresh PR #531 after OpenAPI DX merge
seonghobae Aug 5, 2026
b5bea27
test(auth): require fail-closed parser readiness
seonghobae Aug 5, 2026
6ec3e85
feat(auth): freeze fail-closed runtime and readiness
seonghobae Aug 5, 2026
ea29aae
merge(develop): preserve fail-closed authentication implementation
seonghobae Aug 5, 2026
a2ec3ed
test(security): define bearer protocol and readiness failure edges
seonghobae Aug 5, 2026
61c00ae
fix(security): normalize invalid Unicode secrets as config failures
seonghobae Aug 5, 2026
22452f2
fix(security): honor auth scheme casing and probe failures
seonghobae Aug 5, 2026
b1735ce
test(deployment): require restricted unreleased manifest contracts
seonghobae Aug 6, 2026
e9efa48
fix(deployment): harden the unreleased Kubernetes example
seonghobae Aug 6, 2026
66a2d07
docs(changelog): remove contradictory deployment guidance
seonghobae Aug 6, 2026
e5dc425
docs(operations): align release, auth, and Kubernetes guidance
seonghobae Aug 6, 2026
e304fad
fix(tests): remove unused authentication test import
seonghobae Aug 6, 2026
47f5ced
docs(doctoring): record restricted deployment and release boundaries
seonghobae Aug 6, 2026
da2f26c
fix(security): complete restricted Kubernetes runtime
seonghobae Aug 6, 2026
f647bd8
test(security): enforce restricted deployment contract
seonghobae Aug 6, 2026
926e747
fix(deploy): restore organization GHCR image identity
seonghobae Aug 6, 2026
4f51ff9
fix(test): preserve Python 3.10 deployment metadata checks
seonghobae Aug 6, 2026
2dd7aa6
fix(security): document bounded GHCR registry exception
seonghobae Aug 6, 2026
2d688b3
test(docs): align canonical deployment assertions
seonghobae Aug 6, 2026
30191ec
test(capacity): require bounded default parser admission
seonghobae Aug 6, 2026
1efc918
feat(capacity): default each process to one parse
seonghobae Aug 6, 2026
ac7a538
test(capacity): require configured parser budget
seonghobae Aug 6, 2026
1bb99ea
feat(capacity): load configured parser budget
seonghobae Aug 6, 2026
7a8d2a0
test(capacity): require bounded parser budget values
seonghobae Aug 6, 2026
809682c
test(capacity): cover non-integer direct budgets
seonghobae Aug 6, 2026
923c782
feat(capacity): validate bounded parser budgets
seonghobae Aug 6, 2026
cc0b938
test(capacity): require app-local parser admission limiter
seonghobae Aug 6, 2026
ae0c056
feat(capacity): add application-local concurrency state
seonghobae Aug 6, 2026
57dd887
feat(capacity): bind app-local parser admission state
seonghobae Aug 6, 2026
f8af6c1
test(capacity): require non-waiting admission leases
seonghobae Aug 6, 2026
9cd6382
feat(capacity): implement non-waiting parser leases
seonghobae Aug 6, 2026
5b50070
test(capacity): require pre-body saturation response
seonghobae Aug 6, 2026
925b791
feat(capacity): reject saturated parser work before body reads
seonghobae Aug 6, 2026
056c886
test(capacity): keep saturation contract lint-clean
seonghobae Aug 6, 2026
0accc51
test(capacity): prove lease safety and burst backpressure
seonghobae Aug 6, 2026
0255c53
test(capacity): require operator and standards evidence
seonghobae Aug 6, 2026
6f5d0eb
test(auth): isolate token concurrency from parser capacity
seonghobae Aug 6, 2026
fe1da17
docs(capacity): publish the default process budget
seonghobae Aug 6, 2026
28be99a
docs(capacity): configure compose parser admission
seonghobae Aug 6, 2026
cdcd9d7
docs(capacity): configure Kubernetes parser leases
seonghobae Aug 6, 2026
1d229c8
docs(capacity): explain parser overload behavior
seonghobae Aug 6, 2026
a0a2e22
docs(capacity): record the admission architecture
seonghobae Aug 6, 2026
871f132
docs(capacity): add overload sizing and rollback runbook
seonghobae Aug 6, 2026
e5b8168
docs(capacity): add bounded admission decision record
seonghobae Aug 6, 2026
02a1614
docs(capacity): record bounded parser admission
seonghobae Aug 6, 2026
090979e
docs(security): add parser resource-exhaustion controls
seonghobae Aug 6, 2026
9fde283
test(benchmark): define upload-ingestion evidence contract
seonghobae Aug 6, 2026
8ac6a13
feat(benchmark): add reproducible upload-ingestion matrix
seonghobae Aug 6, 2026
3b516dc
docs(benchmark): add strict upload-ingestion evidence schema
seonghobae Aug 6, 2026
968e2b4
docs(benchmark): record upload measurement methodology
seonghobae Aug 6, 2026
8995dd8
test(benchmark): allow audited custom concurrency reports
seonghobae Aug 6, 2026
b306861
fix(benchmark): validate custom positive concurrency
seonghobae Aug 6, 2026
c4fbda6
test(benchmark): require case-local RSS and true event-loop p95 evidence
seonghobae Aug 6, 2026
eb0e2d3
fix(benchmark): measure case-local RSS and event-loop p95
seonghobae Aug 6, 2026
4b044b9
docs(benchmark): version case-local memory evidence schema
seonghobae Aug 6, 2026
bd7ef15
docs(benchmark): correct RSS and event-loop measurement contract
seonghobae Aug 6, 2026
95b523c
chore(capacity): refresh onto protected develop
seonghobae Aug 11, 2026
2587311
Merge branch 'develop' into feat/bounded-parse-admission
opencode-agent[bot] Aug 22, 2026
8552a68
docs: make NewsDOM landing product-first and license-aware
seonghobae Sep 2, 2026
0ae9f94
docs: remove restricted parser install path
seonghobae Sep 2, 2026
f0edda0
test: enforce commercial parser documentation boundary
seonghobae Sep 2, 2026
2bc9601
docs: refresh NewsDOM public manual landing
seonghobae Sep 2, 2026
35ad4e9
docs: correct organization Pages metadata
seonghobae Sep 2, 2026
042b506
docs(security): use organization advisory route
seonghobae Sep 2, 2026
d29ab19
fix(ci): reconcile dependency owner and documentation contracts
seonghobae Sep 19, 2026
4824d88
chore(repo): remove inherited one-shot repair script
seonghobae Sep 19, 2026
bb7474c
test(benchmark): require reproducible bounded fixtures
seonghobae Sep 19, 2026
a370d2e
test(readiness): reject unapproved legacy parser
seonghobae Sep 19, 2026
f8efe44
test(benchmark): require manifest schema revision
seonghobae Sep 19, 2026
ed4d934
fix(benchmark): bind reports to reproducible host evidence
seonghobae Sep 19, 2026
24b47f7
docs(benchmark): require reproducibility manifest
seonghobae Sep 19, 2026
f244356
fix(readiness): require an approved parser backend
seonghobae Sep 19, 2026
87bdd14
fix(config): keep environment keys ordered
seonghobae Sep 19, 2026
9ab513a
docs(architecture): align immutable runtime settings
seonghobae Sep 19, 2026
96fb2d8
docs(changelog): record approved parser readiness
seonghobae Sep 19, 2026
ae27e4c
docs(install): bind local API shell to loopback
seonghobae Sep 19, 2026
210b7a7
docs(benchmark): document required environment manifest
seonghobae Sep 19, 2026
70d0229
docs(readiness): state approved parser invariant
seonghobae Sep 19, 2026
dd5ebcf
test(docs): align public URL and loopback contracts
seonghobae Sep 26, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -3,5 +3,7 @@ NEWSDOM_AUTH_MODE=required
NEWSDOM_RUNTIME_PROFILE=production
# Generate a high-entropy token in the deployment secret store. Never commit it.
NEWSDOM_API_TOKEN=
# Per-process non-waiting parser admission budget. Scale replicas separately.
NEWSDOM_MAX_CONCURRENT_PARSES=1
Comment thread
seonghobae marked this conversation as resolved.
Resolved
# Optional API-only image override for a bundled or mounted MinerU executable.
# NEWSDOM_MINERU_BIN=/opt/mineru/bin/mineru
131 changes: 95 additions & 36 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,54 +2,113 @@

## Runtime shape

`newsdom-api` is a small service-oriented Python application with a
thin FastAPI entrypoint and explicit separation between request
orchestration, MinerU process execution, and DOM normalization.
`newsdom-api` is a small service-oriented Python application with a thin
FastAPI entrypoint and explicit separation between request security,
process-local admission, MinerU process execution, and DOM normalization. The
same application can run as a standalone service, a sidecar, or a bounded MSA
module behind an API gateway.

## Primary modules

- `src/newsdom_api/main.py` exposes `/health` and `/parse`
through FastAPI.
- `src/newsdom_api/service.py` orchestrates PDF parsing,
temporary files, and response construction.
- `src/newsdom_api/mineru_runner.py` shells out to the MinerU CLI,
collects JSON outputs, and translates runtime or incomplete-output
failures into typed sanitized exceptions.
- `src/newsdom_api/dom_builder.py` converts MinerU `content_list`
blocks plus page model metadata into the canonical NewsDOM response
model.
- `src/newsdom_api/main.py` exposes `/health`, `/ready`, and `/parse` through
FastAPI. Its outer middleware authenticates and admits parser work before the
multipart body is read.
- `src/newsdom_api/config.py` snapshots immutable runtime authentication,
profile, executable, and per-process capacity settings during application
creation.
Comment thread
seonghobae marked this conversation as resolved.
Resolved
- `src/newsdom_api/admission.py` owns the process-local non-waiting parser lease
pool. `ParseAdmissionLimiter` uses `threading.BoundedSemaphore` so excess
release is treated as a programming error rather than silently inflating
capacity.
- `src/newsdom_api/service.py` orchestrates PDF parsing and response
construction.
- `src/newsdom_api/mineru_runner.py` shells out to the MinerU CLI, collects JSON
outputs, and translates runtime or incomplete-output failures into typed
sanitized exceptions.
- `src/newsdom_api/dom_builder.py` converts MinerU `content_list` blocks plus
page model metadata into the canonical NewsDOM response model.
- `src/newsdom_api/schemas.py` defines the public response schema.
- `src/newsdom_api/synthetic.py` and
`src/newsdom_api/equivalence.py` support synthetic fixture
generation and structural comparisons.
- `src/newsdom_api/synthetic.py` and `src/newsdom_api/equivalence.py` support
synthetic fixture generation and structural comparisons.

## Request flow

1. `src/newsdom_api/main.py` receives an uploaded PDF.
2. `src/newsdom_api/service.py` writes the upload to a temporary
workspace and calls MinerU.
3. `src/newsdom_api/mineru_runner.py` resolves the executable, runs
the OCR pipeline, loads generated JSON artifacts, and raises typed
sanitized errors for runtime-unavailable or incomplete-output cases.
4. `src/newsdom_api/dom_builder.py` normalizes OCR blocks into the
canonical response while preserving page-aware structure from
MinerU model metadata.
5. FastAPI returns typed JSON from `src/newsdom_api/schemas.py` and
maps MinerU runtime failures to 503 and incomplete output to 502.
```mermaid
flowchart LR
C[Client] --> A[Raw bearer authentication]
A -->|invalid| U[401 or configuration 503]
A -->|valid| L[Process-local non-waiting lease]
L -->|saturated| R[429 Too Many Requests\nRetry-After: 1]
L -->|admitted| M[Multipart body and upload limits]
M --> P[PDF structural validation]
P --> T[Temporary file]
T --> X[MinerU execution]
X --> D[Canonical DOM normalization]
D --> J[Typed JSON response]
J --> F[Lease release]
X -->|error or cancellation| F
```

1. `src/newsdom_api/main.py` validates the raw Authorization header without
decoding duplicate or non-ASCII credentials into a lossy string.
2. An authenticated `POST /parse` attempts one lease from
`ParseAdmissionLimiter` **before the multipart body** is parsed. The attempt
never waits. Saturation returns `429 Too Many Requests`, `Retry-After: 1`,
and non-cacheable response headers.
3. The route validates MIME type, form values, declared and streamed size, and
PDF structure before parser execution.
4. `src/newsdom_api/service.py` and `src/newsdom_api/mineru_runner.py` execute
MinerU under the configured runtime and error boundaries.
5. `src/newsdom_api/dom_builder.py` normalizes OCR blocks into the canonical
response while preserving page-aware structure from MinerU model metadata.
6. The outer `finally` boundary releases the parser lease after success,
validation failure, backend failure, request cancellation, or another
exception.
7. FastAPI returns typed JSON from `src/newsdom_api/schemas.py` and maps MinerU
runtime failures to 503 and incomplete output to 502.

## Capacity model

`NEWSDOM_MAX_CONCURRENT_PARSES` is immutable after application creation, accepts
`1..128`, and defaults to `1`. The budget is **per process**, not global. A
single-process replica configured with `N` can execute at most `N` MinerU jobs
at once. A deployment with `R` replicas and one serving process in each has a
nominal upper bound of `R × N`; additional gateway, scheduler, CPU, memory,
VRAM, and storage constraints can reduce effective throughput.

The application intentionally does not wait for a parser slot. An internal
queue would already retain request bodies and open connections, hide overload
from the orchestrator, and make memory consumption proportional to demand.
The fixed 429 contract instead keeps queue ownership at a gateway or durable
job service that can implement tenant policy, persistence, cancellation, and
bounded retry semantics.

## Liveness and readiness

`GET /health` proves only that the web process is live. `GET /ready` proves that
fail-closed authentication configuration and the MinerU executable are
available. Parser saturation does not make a process unready: readiness is a
routing eligibility signal, whereas the admission response is instantaneous
load feedback for each request. A future durable asynchronous job API may add
queue-depth readiness policy without changing this synchronous endpoint's
non-waiting contract.

## Supporting systems

- `tests/fixtures` holds synthetic PDFs, JSON baselines, and
provenance notes; private reference inputs stay out of git.
- `tests/fixtures` holds synthetic PDFs, JSON baselines, and provenance notes;
private reference inputs stay out of git.
- `manual/` is the published user manual rendered by MkDocs.
- `.github/workflows/` encodes CI, security scanning, Pages,
release, and image-delivery policy.
- `scripts/release/` builds release manifests and exports GitHub attestation bundles.
- `.github/workflows/` encodes CI, security scanning, Pages, release, and
image-delivery policy.
- `scripts/release/` builds release manifests and exports GitHub attestation
bundles.
- `docs/doctoring/bounded-parser-admission.md` records the resource-exhaustion,
HTTP, caching, implementation, rollback, and APA 7 evidence behind the
admission boundary.

## Delivery boundaries

- `develop` is the integration line for normal feature, fix,
and chore work.
- `develop` is the integration line for normal feature, fix, and chore work.
- `main` is the stable release line that receives tagged releases.
- The service is production-grade only when code, docs, workflows,
and release evidence agree.
- The service is production-grade only when code, docs, workflows, and release
evidence agree.
4 changes: 3 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,13 +20,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- OpenAPI 제목/설명, README, `ArticleNode.headline` 문서를 일반 문서용 (section heading) 표현으로 재구성하여 특정 언어/신문 가정을 소비자에게 노출하지 않도록 함. 응답 스키마 필드는 하위 호환을 위해 변경하지 않음.

### Added
- Added immutable `NEWSDOM_MAX_CONCURRENT_PARSES` process admission with a conservative default of one. Authentication remains first; an authenticated request must acquire a non-waiting lease before the multipart body, temporary-file allocation, PDF validation, or MinerU. Saturation returns fixed `429 Too Many Requests`, `Retry-After: 1`, and no-store cache headers, while every success, error, exception, and cancellation path releases its lease. Docker Compose and Kubernetes examples publish the explicit per-process value and explain replica capacity.
- [CLI] 단일 NewsDOM JSON 파일을 페이지 단위로 분리하는 `tools/split_dom.py` 도구를 추가했습니다.
- [CLI] NewsDOM JSON 파일의 모든 텍스트 내용을 마스킹하여 익명화하는 `tools/anonymize_dom.py` 도구를 추가했습니다.
- `/parse`에 기본 필수 bearer 인증 경계를 추가했습니다. `NEWSDOM_AUTH_MODE=required`, `NEWSDOM_RUNTIME_PROFILE=production`, `NEWSDOM_API_TOKEN`을 명시해야 하며, 인증 비활성화는 격리된 development 프로필에서만 허용됩니다. `/health`는 liveness 전용으로 미인증 상태를 유지하고 `/ready`가 인증 설정과 MinerU 가용성을 함께 검증합니다.
- 서브모듈/사이드카 배포용 `docker-compose.yml`: 필수 production 인증과 secret 주입을 요구하고 healthcheck가 `/ready`를 대상으로 하도록 구성했습니다. Kubernetes 예시는 Restricted Pod Security 설정, 비루트 실행, 권한 상승 금지, 모든 capability 제거, 읽기 전용 root filesystem, 제한된 임시 볼륨, liveness/readiness 분리를 적용합니다.
- [CLI] 파싱된 NewsDOM JSON에서 순수 텍스트 데이터를 추출하여 텍스트 파일 또는 stdout으로 출력하는 `tools/extract_text.py` 도구를 추가했습니다.

### Security
Comment thread
seonghobae marked this conversation as resolved.
Resolved
- Bounded active parser work prevents one process from starting unbounded MinerU jobs under burst traffic. Excess requests fail before body reads with the RFC 6585 429 contract and RFC 9111 no-store semantics; gateway-level tenant quotas and durable job queues remain separate controls.
- `/parse` authentication is now immutable per application instance and fails closed before multipart body parsing when required configuration is missing. Hostile missing, invalid, Unicode, oversized, and duplicated Authorization headers return one non-sensitive response.
- Added unauthenticated `/ready` traffic readiness that combines authentication configuration with MinerU executable availability while `/health` remains liveness-only.
- Hardened the Kubernetes deployment example with a restricted namespace policy, explicit non-root UID/GID, `RuntimeDefault` seccomp, disabled privilege escalation, dropped Linux capabilities, a read-only root filesystem, and bounded writable runtime volumes.
Expand Down Expand Up @@ -64,7 +66,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Changed

- Improved OpenAPI metadata for the `/parse` endpoint by documenting 415, 502, and 503 error responses.
- Improved OpenAPI metadata for the `/parse` endpoint by documenting 415, 429, 502, and 503 error responses.

## [0.2.0] - 2026-04-24

Expand Down
Loading
Loading