Auto-assembly using Croissant ML metadata—separation of semantic schema and data element metadata

[slobentanzer]

flowchart TD

%% ---------------------------
%% SUBGRAPHS TO SHOW LAYERS
%% ---------------------------

subgraph UserLayer[User Layer]
    Ud[User defines ETL]
    U[User selects concepts]
end

subgraph SemanticLayer[Semantic Layer]
    SC[BioCypher schema]
end

subgraph MetadataLayer[Data Element Metadata Layer]
    DE_existing[Existing metadata in adapters]
    DE_new[New BioCroissant based metadata]
end

subgraph AdapterLayer[Adapter Layer]
    A_existing[Manual adapters]
    A_gen[Generated adapters from metadata]
end

subgraph AssemblyLayer[Assembly and Build Layer]
    AM_existing[Current schema plus adapter matching]
    AM_new[Future automatic assembly]
end

subgraph GraphLayer[Graph Construction Layer]
    GC[BioCypher graph builder]
end


%% ---------------------------
%% FLOWS
%% ---------------------------

U --> SC
Ud --> DE_existing
U --- Ud

SC --> AM_existing
SC --> AM_new

DE_existing --> A_existing
DE_new --> A_gen
DE_new --> AM_new

A_existing --> GC
A_gen --> GC

AM_existing --> GC
AM_new --> GC

%% ---------------------------
%% COLOUR STYLING
%% ---------------------------

%% Legacy components (keep)
style SC fill:#dbeafe,stroke:#1e40af,stroke-width:1px
style GC fill:#dbeafe,stroke:#1e40af,stroke-width:1px

%% Deprecated components (phase out)
style Ud fill:#fee2e2,stroke:#b91c1c,stroke-width:1px
style A_existing fill:#fee2e2,stroke:#b91c1c,stroke-width:1px
style DE_existing fill:#fee2e2,stroke:#b91c1c,stroke-width:1px
style AM_existing fill:#fee2e2,stroke:#b91c1c,stroke-width:1px

%% New additions
style U fill:#dcfce7,stroke:#15803d,stroke-width:1px
style DE_new fill:#dcfce7,stroke:#15803d,stroke-width:1px
style A_gen fill:#dcfce7,stroke:#15803d,stroke-width:1px
style AM_new fill:#dcfce7,stroke:#15803d,stroke-width:1px

Loading

High-Level Refactor Plan: Clean Separation of Semantics and Data-Element Metadata in BioCypher

Goal

Make BioCypher easier to use and more maintainable by separating semantic modelling (ontology-based) from data recording metadata (technical representation)

Achieve a workflow where users select domain concepts, and BioCypher automatically assembles the ETL, metadata validation, and graph construction behind the scenes.

Current State

BioCypher Schema (semantic layer)

• Defines domain concepts (Gene, Variant, Disease, etc.)
• Provides ontology mappings
• Defines graph node/edge types

Graph Builder

• Takes semantic definitions + adapter output
• Constructs Neo4j/RDF/NetworkX graphs

Manual Adapters (where needed)

• Continue to work as they are

The Problem Today

• Adapters implicitly contain data element metadata (names, units, datatypes).
• Schema configuration must be manually aligned with adapter outputs.
• Users must understand both ETL structure and semantics → high cognitive load.
• Metadata definitions in newer projects (e.g., Jinja templates) mix semantics and structure.

This creates tight coupling and repeated effort.

The New Architecture (Additions)

A dedicated metadata layer (BioCroissant style)

A structured metadata registry describing:

• variable names
• datatypes, units, permissible values
• measurement/provenance
• dataset boundaries

This is NOT semantic. It is purely technical representation.

BioCypher does not interpret this as ontology.
It just uses it to produce validated structured data.

Automatic Adapter Generation from Metadata

Using a templating system (e.g., Jinja), metadata now produces:

• ETL code
• validation routines
• dataset readers
• standardised data record objects

Users no longer write ETL unless they want to.

Automatic Assembly Layer

A new component that:

1. Takes user-selected semantic concepts
2. Finds which datasets contain relevant data elements (from metadata)
3. Automatically loads or generates the required adapters
4. Produces an ETL pipeline that feeds the graph builder

User only selects concepts; BioCypher handles everything else.

Clean Separation of Concerns

Semantics (BioCypher schema)

• Defines “what exists in the knowledge graph”
• Ontology classes and relations only
• NO datatypes, NO file paths, NO variable definitions

Metadata (BioCroissant)

• Defines “how data is represented in files”
• Technical attributes, not conceptual ones
• Drives ETL generation, not schema design

Adapters

• Consume metadata
• Produce normalised, validated records
• Bind results to semantic classes via schema

Graph Builder

• Remains unchanged
• Receives semantic entities + metadata-rich records

Components Being Deprecated

These are phased out incrementally:

• User-defined ETL: replaced by metadata-driven adapter generation
• Manual schema–adapter sync: replaced by automatic assembly layer
• Metadata hidden inside code: replaced by explicit metadata registry

These remain for compatibility but are no longer the recommended pathway.

Expected Benefits

For Users

• Only need to know domain concepts
• No need to learn metadata formats or understand the ETL
• No schema–adapter synchronisation
• More predictable graph builds from heterogeneous data lakes

For Developers

• Stronger modularity: schema ≠ metadata ≠ code
• Cleaner responsibilities and testability
• Easier debugging (metadata validation layer)
• Extensible: aligns with BioCroissant, RO-Crate, Schema.org, etc.

For Future Integrations (BioContext, LLM tooling)

• Agents can introspect metadata
• Automatic tool selection becomes feasible
• Runtime adaptivity in ETL

Summary

Target is a three-layer architecture:

1. Semantic Layer
   • Existing schema, unchanged responsibility
   • Should not contain technical metadata
2. Metadata Layer (new)
   • BioCroissant-inspired
   • Defines all technical data structures
   • Drives adapter generation
3. Assembly Layer (new)
   • Takes user semantic intent
   • Chooses datasets + adapters
   • Sends assembled pipeline to the graph builder

Adapters and graph builder remain compatible and largely unchanged, but their inputs are now better defined and automatically produced.

[slobentanzer]

Hi @kpto, happy new year! Would be interested in your opinion on this.