Skip to content

Initializer storageUri is silently overridden by user-provided STORAGE_URI env var #3955

Description

@mimosapudical

What happened?

When a user sets both initializer.dataset.storageUri (or initializer.model.storageUri)
and a STORAGE_URI entry in initializer.dataset.env / initializer.model.env, the
user-provided env var silently wins, even though storageUri is the structured,
documented field for specifying the source.

In pkg/runtime/framework/plugins/jobset/builder.go, the initializer container's
env vars are built in this order:

if storageUri != nil {
    apply.UpsertEnvVars(env, STORAGE_URI=<storageUri>)
}
apply.UpsertEnvVars(env, userEnv...)

Since UpsertEnvVars replaces any existing entry with the same name, if userEnv
also contains STORAGE_URI, it overwrites the value derived from storageUri
with no error, warning, or indication to the user.

Example:

initializer:
  dataset:
    storageUri: "hf://tatsu-lab/alpaca"
    env:
      - name: STORAGE_URI
        value: "hf://other/repo"

The TrainJob spec appears to declare hf://tatsu-lab/alpaca as the dataset source,
but the initializer container actually receives hf://other/repo.

What did you expect to happen?

Either:

  • storageUri should take precedence as the structured source of truth, and a
    conflicting STORAGE_URI in env should be rejected/warned at admission time, or
  • If env overriding storageUri is intentional (e.g. as an escape hatch), this
    precedence should be explicitly documented, so the TrainJob spec doesn't visually
    misrepresent what's actually being downloaded.

Currently neither is the case — the override happens silently, and I couldn't find
existing docs, KEPs, or tests that establish this as intended behavior.

Environment

  • Kubeflow Trainer version: (fill in)
  • Relevant file: pkg/runtime/framework/plugins/jobset/builder.go

Impacted by this bug?

Give it a 👍 We prioritize the issues with most 👍

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Status
    Needs Triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions