Skip to content

feat(core): add TenderDataScanner for architectural bidding and geospatial coordinates - #27

Closed
jmcaamanog wants to merge 1 commit into
InditexTech:mainfrom
jmcaamanog:feat/tender-coordinates-scanner
Closed

jmcaamanog wants to merge 1 commit into
InditexTech:mainfrom
jmcaamanog:feat/tender-coordinates-scanner

Conversation

@jmcaamanog

Copy link
Copy Markdown
Contributor

Description

This pull request introduces TenderDataScanner into cerbia-core, providing specialized threat detection and data leakage prevention for technical architectural tenders, construction bidding data, and high-precision geospatial coordinates.

In commercial architecture, store rollout programs, and facilities engineering, AI models and autonomous agents frequently process project documentation that may contain unreleased tender prices, contractor margin structures, or coordinates of confidential land acquisitions.

Key Capabilities

  1. FIEBDC-3 / BC3 Standard Bidding Record Detection:
    • Detects standard European architectural exchange records (~V database headers, ~K indirect cost coefficients, ~C concept codes and unit prices, ~D decomposition trees, ~M detailed measurement units).
    • Prevents proprietary contractor cost structures and price breakdowns from exfiltration.
  2. Construction Budget & Margin Extraction:
    • Detects formalized tender budget totals: PEM (Presupuesto de Ejecución Material), PBL (Presupuesto Base de Licitación), and contractor margins (B.I. Beneficio Industrial, G.G. Gastos Generales).
  3. Official Technical Procurement Dockets:
    • Detects Spanish and EU public/private procurement docket identifiers (EXP-..., LIC-..., and TED / DOUE notice numbers).
  4. Sub-Meter Geospatial Coordinates:
    • Detects high-precision geographic coordinate pairs (decimal degrees with $\ge 4$ decimal places, sexagesimal DMS notation, and projected UTM/ETRS89 coordinates) to protect confidential unannounced commercial sites and logistics assets.

Testing & Verification

  • Unit Tests: 11 targeted unit tests in packages/cerbia-core/tests/unit/cerbia/core/scanners/tender/test_scanner.py verifying full pattern matching, span offsets, configurable detector toggles, and risk scoring.
  • Factory Integration: Registered and tested in ScannerFactory (test_scanner.py).
  • Core Test Suite: All 722 unit tests in cerbia-core pass with 96% overall coverage.

Quality & Contribution Standards

  • DCO Signed-off: Commit signed with Signed-off-by: jmcaamanog <jose.caamano@live.com>.
  • Cryptographic SSH Signature: Verified with ed25519 key.
  • REUSE 3.3 Compliant: All files include SPDX headers; reuse lint passes 387/387 files.
  • Code Quality: Passes ruff check and ruff format.
  • Testing: 100% test pass rate across cerbia-core.

…atial coordinates

Signed-off-by: jmcaamanog <jose.caamano@live.com>
@jmcaamanog
jmcaamanog requested a review from a team as a code owner October 14, 2026 08:26
@signoff-validator

Copy link
Copy Markdown

Contribution License Agreement - Valid 🟢

All commits contain valid signoff. As a confirmation, remember that you are agreeing to our Contribution License Agreement.

@sergiobe-itx

Copy link
Copy Markdown
Contributor

Thanks for all these contributions to Cerbia!

I don’t think this scanner fits in Cerbia, since it is a generic library, and this scanner covers a very specific domain with its own rules about confidential data.

I think the best home for this work is an independent extension package, outside this repository. Cerbia already supports custom scanners through fully qualified import paths, so users could install your package and configure it without any changes to Cerbia.

Since I already reviewed the implementation, I’d also like to share the findings in packages/cerbia-core/src/cerbia/core/scanners/tender/_constants.py:

Location Finding Suggestion
Regex performance Lines 12, 19–22, 41 and 45 Several patterns become much slower with long whitespace input. For example, "PEM" + " " * n + "!" took about four times longer when the input length doubled. Some tests took several seconds with around 32 KB of input. This can cause a denial of service when scanning untrusted text. Rewrite the whitespace handling to avoid repeated backtracking. Add regression tests with these inputs and a time limit.
Invalid coordinates Lines 41 and 45 Invalid coordinates such as 43°21'44.4"N 999°24'40.4"W and UTM Zone 99Z X: 547890 Y: 4799120 are accepted with a risk score of 0.75. Validate coordinate ranges, UTM zones, and the limits at 90° latitude and 180° longitude.
Missed UTM coordinates Line 45 UTM Zone 31N X: 500000.123 Y: 553000.456 returns no finding because the pattern requires exactly seven integer digits for northing. Valid northings can have fewer digits. Support valid northing values without requiring a fixed number of digits. Add a scanner-level regression test.
Partial matches Lines 29 and 37 EXP-2026/00142/OBR matches only EXP-2026/00142. Also, 43.3623, 8.4112e9 matches only 43.3623, 8.4112, which treats part of a larger number as a coordinate. Check the full token boundaries. Add tests that verify the exact text covered by each returned span.
Precision claims Lines 35–45 Four decimal places do not mean sub-meter resolution. A latitude step of 0.0001° is about 11 metres. Whole DMS seconds are also not sub-meter. Adjust the precision claims and document the supported formats and limits.
Confidentiality claims Scanner rationale and documentation Finding a budget, public procurement identifier, or coordinate does not prove that the data is confidential. Describe the results as potential tender or location data. Let each application define its own disclosure policy.

@jmcaamanog

Copy link
Copy Markdown
Contributor Author

Hi @sergiobe-itx!

Thank you very much for the thoughtful explanation and the detailed technical review!

I completely agree with your architectural decision: CerbIA is designed to be a lightweight, general-purpose security and threat-detection perimeter, so domain-specific scanners (like construction / architectural bidding data and geospatial coordinates) fit much better as an independent extension package using CerbIA's fully qualified custom scanner import mechanism.

Your technical findings on regex backtracking under long whitespace, coordinate range validation (lat/long/UTM limits and variable northing digits), token boundary matching, and clarifying precision/confidentiality claims are invaluable. I will take all these points into account when refining and publishing the standalone extension package.

Thanks again for the guidance and for reviewing all the contributions! 🙌

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants