new: dev: [ci] initial auto-update list workflow

[achrinza]

This is a draft PR for a CI workflow that auto-updates the lists on a weekly basis and also ad-hoc list updates via workflow dispatch.

How it's Setup

• Auto-updates are done weekly and can also be triggered manually
• Updates are surfaced as PRs for review, no direct push to the default branch
• SLSA L3 provenance attestations are created as part of the build workflow
  When combined with the workflow artifact, this is useful for tracing back the CI state+environment which generated a specific iteration of a warninglist. This helps with reproducibility (see TODOs below).

Additional configuration needed:

• The workflow expects CLOUDFLARE_RADAR_API_TOKEN to be configured as a repository secret

Other Misc. Changes

These changes are complementary to the auto-update workflow:

• generator.py
  • Tweaked to not update the warninglists' version when the list itself has not changed.
  • Now throws when the request returns non-200 error code (mitigates "successful", but wrong list generation)
• Dependency management
  • pyproject.toml replaces the older requirements.txt
  • A pylock.toml file is used to improve the resiliency of the pipeline (from supply chain attacks and unwanted breaking changes) by allowing dependency upgrades to be controlled
    • Ideally this would be paired with an auto-update tool like Renovatebot or Dependabot
    • This does not prevent consumers from using latest versions of dependencies as concerned by Project dependencies may have API risk issues #234 (comment). They can do so with pip install -r pyproject.toml. For consumers who want to have a stable dependency tree, they can do pip install -r pylock.toml --no-deps.
  • Similarly, the Github Actions actions are pinned by hash to mitigate supply chain attacks

Breaking changes

Min. Python upgraded to v3.10

• pip enforces Python minimum version requirements set by dependencies through pyproject.toml (which is 3.10)
• 3.10 is the earliest non-EOL Python version.

TODOs

• Add pull_request validation
  To gate pull requests such that it cannot break the CI pipeline.
• Simplify attestation
  • Limit attestations for script-generated files
    It's a bit pointless to attest a file that never gets touched by the CI pipeline
  • Collapse attestation workflow into main workflow
    • Trusted, reusable attestation workflow eases management when shared across repos. This doesn't apply for single-repos like here.
    • A common workflow file also means that attestations can be done against the whole workflow file's digest, and thus it's possible to programmatically detect when the workflow file changes without needing the repo's Git history.
• Improve attestation
  • Extend workflow artifact expiry to 90 days
    It's set to 1 day to reduce clutter during testing. However, the usefulness of the SLSA provenance depends on having access to the tmp directory generated during build time. This is stored in the auto-updated-lists artifact.
  • Make workflow artifact unique and non-overwriting
    Same reason as above; We should ideally retain all copies of generate workflow artifacts for each run.

---

Example auto-update PR: achrinzafork#2

Example attestation: https://github.com/achrinzafork/misp-warninglists/attestations/40266806

Example attestation verification:

$ gh attestation verify \
  --signer-workflow=achrinzafork/misp-warninglists/.github/workflows/update-lists-attest.yaml@refs/heads/main \
  --repo=achrinzafork/misp-warninglists \
  --limit=1 \
  ./lists/cloudflare/list.json
Loaded digest sha256:ac58196b0327ec6109a3d4b1117f9e079e7172f965c942f50206aff0d97c4851 for file://lists/cloudflare/list.json
Loaded 1 attestation from GitHub API

The following policy criteria will be enforced:
- Predicate type must match:................ https://slsa.dev/provenance/v1
- Source Repository Owner URI must match:... https://github.com/achrinzafork
- Source Repository URI must match:......... https://github.com/achrinzafork/misp-warninglists
- Subject Alternative Name must match regex: ^https://github.com/achrinzafork/misp-warninglists/.github/workflows/update-lists-attest.yaml@refs/heads/main
- OIDC Issuer must match:................... https://token.actions.githubusercontent.com

✓ Verification succeeded!

The following 1 attestation matched the policy criteria

- Attestation #1
  - Build repo:..... achrinzafork/misp-warninglists
  - Build workflow:. .github/workflows/update-lists.yaml@refs/heads/main
  - Signer repo:.... achrinzafork/misp-warninglists
  - Signer workflow: .github/workflows/update-lists-attest.yaml@refs/heads/main

closes: #102