Add ml-lineage datapack: warehouse to production ML models with column-level lineage - #224
Open
malharlakdawala wants to merge 1 commit into
Open
malharlakdawala wants to merge 1 commit into
malharlakdawala wants to merge 1 commit into
Conversation
25 entities: 7 Snowflake datasets with 21 column-level lineage mappings, 2 mlFeatureTables with 7 mlFeatures, and 3 mlModels in 3 mlModelGroups. Neither bootstrap nor showcase-ecommerce contains ML entities, so there is no sample data for mlModel/mlFeature lineage or the DerivedFrom/Consumes/ MemberOf relationships. The graph is shaped so column-level lineage is discriminating: different raw columns reach one, none, or all three models.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an
ml-lineagedatapack: a small NYC-taxi-shaped warehouse wired all the way throughto production ML models, so the ML half of the metadata model has sample data to exercise.
Neither shipped datapack currently contains ML entities.
bootstrapis datasets, dashboards,users and tags;
showcase-ecommerceis a rich BI/governance graph across Snowflake, Looker,PowerBI and Tableau. Nothing seeds
mlModel,mlModelGroup,mlFeatureTableormlFeature,which makes the ML lineage features and the
DerivedFrom/Consumes/MemberOfrelationships awkward to demo or test against.
What it contains
25 entities, 32 aspects, 68 KB.
schemaMetadataupstreamLineage.fineGrainedLineages, FIELD_SET → FIELDmlFeatureTable+ 7mlFeaturesourcespointing at the feature datasetsmlModelin 3mlModelGroupmlFeatureswired to the featuresThe graph is deliberately shaped so column-level lineage is discriminating rather than
uniform — some raw columns reach a model and some do not:
That makes it useful for testing impact analysis: a table-level query from
raw_paymentsreaches all three models, while the column-level edges reach one, none, or all depending on
which column you start from.
One feature,
tip_volatility, is intentionally not named after its source column(
trip_features.tip_variance), because feature stores rename and a fixture where everyfeature name matches its column hides that case.
Format
Follows the
showcase-ecommercelayout:index.jsonwith aversionand an orderedfilesarray, flat file paths,
wait_for_completionon the two stages later files depend on.Loads with
datahub datapack load ml-lineageonce registered.Registry entry
Registration lives in the main repo, so this PR only adds the content. Happy to open the
matching
registry.jsonPR againstdatahub-project/datahubonce this merges and the rawURL resolves — flagging rather than opening it now so the registry never points at a 404:
Testing
Loaded into a local DataHub Core v1.7.0 quickstart. All 25 entities resolve in the UI, the
column-level lineage renders in the dataset lineage column view, and GraphQL
searchAcrossLineagetraversesdataset → mlFeature → mlModel → mlModelGroupwith theexpected
degreevalues (1 through 4) and full paths.