Skip to content

BPMN eval criteria reject valid SDK preview artifacts #2908

Description

@tmatup

The current Maestro BPMN coder-eval criteria encode two live-authoring representations too narrowly:

  • skill-bpmn-hitl-rpa-wrappers, skill-bpmn-simple-approval-bpmn, and skill-bpmn-e2e-wiki-pageviews hard-code nested project paths, while the Flow SDK preview materializes the same project files at the artifact root. Their semantic checkers already discover the BPMN recursively and validate the files beside it.
  • check_simple_approval_bpmn.py treats the display name as the only variable identity. SDK 3.27.6 correctly emits canonical IDs (expenseId, amount, decision) with descriptive display names, so a structurally valid process is rejected.

Fresh matched evidence from ad-hoc run adhoc-2026-08-30_04-43-27 reproduced both failures. The preview HITL/RPA artifact passes its complete semantic checker and fails only the nested path criterion. The preview simple-approval artifact contains all three required variable IDs and otherwise satisfies the process contract. The preview WikiPageviews artifact passes its semantic checker but its root BPMN and operate.json fail three path-only criteria.

Make the criteria author-loop neutral: accept root, direct-project, and solution-wrapper materializations; recognize canonical variable IDs while retaining legacy name compatibility; add positive and negative coverage. Do not relax any BPMN semantic requirement.

🤖 Generated with Claude Code
Co-Authored-By: Claude

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions