What happened?
When a user sets both initializer.dataset.storageUri (or initializer.model.storageUri)
and a STORAGE_URI entry in initializer.dataset.env / initializer.model.env, the
user-provided env var silently wins, even though storageUri is the structured,
documented field for specifying the source.
In pkg/runtime/framework/plugins/jobset/builder.go, the initializer container's
env vars are built in this order:
if storageUri != nil {
apply.UpsertEnvVars(env, STORAGE_URI=<storageUri>)
}
apply.UpsertEnvVars(env, userEnv...)
Since UpsertEnvVars replaces any existing entry with the same name, if userEnv
also contains STORAGE_URI, it overwrites the value derived from storageUri —
with no error, warning, or indication to the user.
Example:
initializer:
dataset:
storageUri: "hf://tatsu-lab/alpaca"
env:
- name: STORAGE_URI
value: "hf://other/repo"
The TrainJob spec appears to declare hf://tatsu-lab/alpaca as the dataset source,
but the initializer container actually receives hf://other/repo.
What did you expect to happen?
Either:
storageUri should take precedence as the structured source of truth, and a
conflicting STORAGE_URI in env should be rejected/warned at admission time, or
- If
env overriding storageUri is intentional (e.g. as an escape hatch), this
precedence should be explicitly documented, so the TrainJob spec doesn't visually
misrepresent what's actually being downloaded.
Currently neither is the case — the override happens silently, and I couldn't find
existing docs, KEPs, or tests that establish this as intended behavior.
Environment
- Kubeflow Trainer version: (fill in)
- Relevant file:
pkg/runtime/framework/plugins/jobset/builder.go
Impacted by this bug?
Give it a 👍 We prioritize the issues with most 👍
What happened?
When a user sets both
initializer.dataset.storageUri(orinitializer.model.storageUri)and a
STORAGE_URIentry ininitializer.dataset.env/initializer.model.env, theuser-provided env var silently wins, even though
storageUriis the structured,documented field for specifying the source.
In
pkg/runtime/framework/plugins/jobset/builder.go, the initializer container'senv vars are built in this order:
Since
UpsertEnvVarsreplaces any existing entry with the same name, ifuserEnvalso contains
STORAGE_URI, it overwrites the value derived fromstorageUri—with no error, warning, or indication to the user.
Example:
The TrainJob spec appears to declare
hf://tatsu-lab/alpacaas the dataset source,but the initializer container actually receives
hf://other/repo.What did you expect to happen?
Either:
storageUrishould take precedence as the structured source of truth, and aconflicting
STORAGE_URIinenvshould be rejected/warned at admission time, orenvoverridingstorageUriis intentional (e.g. as an escape hatch), thisprecedence should be explicitly documented, so the TrainJob spec doesn't visually
misrepresent what's actually being downloaded.
Currently neither is the case — the override happens silently, and I couldn't find
existing docs, KEPs, or tests that establish this as intended behavior.
Environment
pkg/runtime/framework/plugins/jobset/builder.goImpacted by this bug?
Give it a 👍 We prioritize the issues with most 👍