Skip to content

README.md

Latest commit

 

History

History
118 lines (83 loc) · 3.14 KB

README.md

File metadata and controls

118 lines (83 loc) · 3.14 KB
Raw

Ciberlab Report Generator - Batch Upload (Server Runbook)

Objective

This process:

  1. Lists JSON/ and PDF/ entries in the Hugging Face dataset.
  2. Detects pending samples (JSON files without a PDF).
  3. Downloads JSON files in batches, generates PDFs, and uploads them to Hugging Face.
  4. Attempts to hydrate metadata in MariaDB (if DB credentials are provided).

Dependencies

Minimum requirements to run end-to-end:

  • Docker installed on the server.
  • Outbound Internet access from the container (OpenAI, VirusTotal, and Hugging Face).
  • Required environment variables configured (OPENAI_API_KEY, HF_TOKEN, VT_API_KEY_1).
  • If you want DB writes: connectivity to MariaDB host + valid credentials.

Quick Server Deployment

Run from the repository root.

1. Prepare .env

cp batch_upload/.env.example batch_upload/.env

Edit batch_upload/.env and fill in at least:

  • OPENAI_API_KEY
  • HF_TOKEN (or HUGGINGFACE_TOKEN)
  • VT_API_KEY_1

If you also want DB hydration, fill in:

  • DB_HOST
  • DB_PORT
  • DATABASE
  • MARIADB_CIBERLAB_USER
  • MARIADB_CIBERLAB_PASSWORD

2. Build

docker build -t ciberlab-report-generator-batch -f batch_upload/Dockerfile .

3. Run

docker run --rm --env-file batch_upload/.env ciberlab-report-generator-batch

Recommended Run with Volumes (host-visible logs and artifacts)

docker run --rm \
  --env-file batch_upload/.env \
  -v "$PWD/json:/data/input" \
  -v "$PWD/pdf:/data/output" \
  -v "$PWD/tmp:/app/tmp" \
  -e INPUT_PATH_DEFAULT=/data/input \
  -e OUTPUT_PATH_DEFAULT=/data/output \
  ciberlab-report-generator-batch

Optional Helper Script

You can also use:

batch_upload/batch-upload.sh build
batch_upload/batch-upload.sh run

With volumes:

batch_upload/batch-upload.sh run-mount

If the DB is on a different Docker network, define DOCKER_NETWORK:

DOCKER_NETWORK=app_default batch_upload/batch-upload.sh run

Key Variables

  • BATCH_SIZE: batch size (download/processing/upload).
  • AI_MODEL: OpenAI model to use.
  • MODE: execution mode (PRO, etc.).
  • LOG_LEVEL: DEBUG, INFO, WARNING, ERROR.
  • TMP_PATH: directory for logs and reportGeneratorResults_*.json.

Preproduction Checklist

  • batch_upload/.env does not contain test secrets.
  • HF_TOKEN has write permissions for the target dataset.
  • OpenAI and VirusTotal limits/quotas reviewed for expected volume.
  • Docker networking validated toward MariaDB (if applicable).
  • Output directories (json/, pdf/, tmp/) have correct permissions.

Important Operational Notes

  • If DB credentials are missing, the process still runs (generates/uploads PDFs) but logs DB hydration errors.
  • If PDF upload to Hugging Face fails, upload_errors_*.json is generated and the process stops.
  • Default profile for this entrypoint is batch_uploader.

Quick Troubleshooting

  • OpenAI error on startup: check OPENAI_API_KEY.
  • Upload to Hugging Face fails: check HF_TOKEN and dataset permissions.
  • DB hydration fails: check DB_HOST, credentials, and Docker network.
  • No pending JSON files found: matching PDFs may already exist in the dataset.