-
Notifications
You must be signed in to change notification settings - Fork 10
Add workspace root kep #10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
mjudeikis
wants to merge
3
commits into
main
Choose a base branch
from
workspaceRoot.kep
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,311 @@ | ||||||||||||||
| # Workspace Root (Forest Support) | ||||||||||||||
|
|
||||||||||||||
| ## Summary | ||||||||||||||
|
|
||||||||||||||
| Introduce a `WorkspaceRoot` API object to enable forest-type workspace hierarchies in kcp, | ||||||||||||||
| allowing multiple independent workspace trees to coexist. This addresses security concerns | ||||||||||||||
| with the current single-root structure where organizational hierarchy is exposed to all users, | ||||||||||||||
| and provides better tenant isolation by giving each organization its own opaque root identifier. | ||||||||||||||
|
|
||||||||||||||
| Currently, users navigating kcp workspaces can infer organizational structure: | ||||||||||||||
| ``` | ||||||||||||||
| root:orgs:company-a:team1:project | ||||||||||||||
| root:orgs:company-b:team2:project | ||||||||||||||
|
mjudeikis marked this conversation as resolved.
Outdated
|
||||||||||||||
| ``` | ||||||||||||||
|
|
||||||||||||||
| With WorkspaceRoot, each organization gets its own isolated tree with an opaque identifier: | ||||||||||||||
| ``` | ||||||||||||||
| a1b2c3d4:team1:project # company-a's tree | ||||||||||||||
| x9y8z7w6:team2:project # company-b's tree | ||||||||||||||
|
mjudeikis marked this conversation as resolved.
Outdated
|
||||||||||||||
| ``` | ||||||||||||||
|
|
||||||||||||||
| ## Motivation | ||||||||||||||
|
|
||||||||||||||
| ### Current State Problems | ||||||||||||||
|
|
||||||||||||||
| 1. **Hierarchy Exposure**: The current single-root structure (`root:orgs:...`) exposes the | ||||||||||||||
| organizational hierarchy to all users. Even without access, users can guess names, based | ||||||||||||||
| on naming conventions. | ||||||||||||||
|
mjudeikis marked this conversation as resolved.
Outdated
|
||||||||||||||
|
|
||||||||||||||
| 2. **Predictable Paths**: Organization names in workspace paths are predictable. Knowing one | ||||||||||||||
| organization exists (e.g., `root:orgs:acme`) allows guessing siblings (`root:orgs:contoso`). | ||||||||||||||
| This is similar to how knowing one AWS account ID allows guessing others. | ||||||||||||||
|
|
||||||||||||||
| 3. **Internal Structure Leakage**: The `root:orgs` path structure is an internal system mechanic | ||||||||||||||
| that exposes implementation details to consumers. Organizations should not need to know they | ||||||||||||||
| exist under a shared `root:orgs` prefix. | ||||||||||||||
|
|
||||||||||||||
| 4. **Navigation Leakage**: Users can walk up the hierarchy using `ws use ..` and discover | ||||||||||||||
| sibling workspaces, even if they receive permission denied errors. While this can be mitigated with rbac, | ||||||||||||||
| but its very easy to misconfigure and leave gaps. | ||||||||||||||
| ```bash | ||||||||||||||
| $ kubectl ws use :root:orgs:bob | ||||||||||||||
| Error: the server has asked for the client to provide credentials # 401 - exists! | ||||||||||||||
|
|
||||||||||||||
| $ kubectl ws use :root:orgs:bob1 | ||||||||||||||
| Error: access to workspace "root:orgs:bob1" denied # 403 - doesn't exist | ||||||||||||||
| ``` | ||||||||||||||
|
mjudeikis marked this conversation as resolved.
Outdated
|
||||||||||||||
|
|
||||||||||||||
|
mjudeikis marked this conversation as resolved.
Outdated
|
||||||||||||||
| ### Goals | ||||||||||||||
|
|
||||||||||||||
| 1. Enable creation of independent workspace trees (forest structure) with opaque root identifiers. | ||||||||||||||
|
mjudeikis marked this conversation as resolved.
Outdated
|
||||||||||||||
| 2. Provide tenant isolation where organizations cannot infer existence of sibling organizations. | ||||||||||||||
| 3. Maintain compatibility with existing workspace hierarchy mechanics within each tree. | ||||||||||||||
| 4. Support WorkspaceType inheritance and WorkspaceAuthenticationConfiguration within trees. | ||||||||||||||
|
mjudeikis marked this conversation as resolved.
Outdated
|
||||||||||||||
| 5. Keep the API simple and focused on tree orchestration. | ||||||||||||||
|
|
||||||||||||||
| ### Non-Goals | ||||||||||||||
|
|
||||||||||||||
| 1. Advanced scheduling across shards (basic scheduling only). | ||||||||||||||
| 2. Replacing the existing workspace hierarchy model - this extends it. | ||||||||||||||
| 3. Cross-tree workspace references or mounts (each tree is isolated). This is out of scope for this KEP but | ||||||||||||||
| technically possible in the future if needed. | ||||||||||||||
|
mjudeikis marked this conversation as resolved.
Outdated
|
||||||||||||||
| 4. Automatic migration of existing workspaces to new trees. | ||||||||||||||
| 5. Facade/translation layer - these are real separate roots, not path rewriting. | ||||||||||||||
|
|
||||||||||||||
| ## Proposal | ||||||||||||||
|
|
||||||||||||||
| ### WorkspaceRoot API | ||||||||||||||
|
|
||||||||||||||
| Introduce a new API object `WorkspaceRoot` in the `tenancy.kcp.io` API group: | ||||||||||||||
|
mjudeikis marked this conversation as resolved.
Outdated
|
||||||||||||||
|
|
||||||||||||||
| ```go | ||||||||||||||
| // WorkspaceRoot defines an independent workspace tree root. | ||||||||||||||
| // Creating a WorkspaceRoot provisions a new LogicalCluster that serves | ||||||||||||||
| // as the root of an isolated workspace hierarchy. | ||||||||||||||
| type WorkspaceRoot struct { | ||||||||||||||
|
mjudeikis marked this conversation as resolved.
Outdated
|
||||||||||||||
| metav1.TypeMeta `json:",inline"` | ||||||||||||||
| metav1.ObjectMeta `json:"metadata,omitempty"` | ||||||||||||||
|
|
||||||||||||||
| Spec WorkspaceRootSpec `json:"spec,omitempty"` | ||||||||||||||
| Status WorkspaceRootStatus `json:"status,omitempty"` | ||||||||||||||
| } | ||||||||||||||
|
|
||||||||||||||
| type WorkspaceRootSpec struct { | ||||||||||||||
| // Name is the optional human-readable name for this root. | ||||||||||||||
| // If not provided, a random base36 identifier will be generated. | ||||||||||||||
| // +optional | ||||||||||||||
| Name string `json:"name,omitempty"` | ||||||||||||||
|
mjudeikis marked this conversation as resolved.
Outdated
|
||||||||||||||
|
|
||||||||||||||
| // Type references a WorkspaceType that defines the initial configuration | ||||||||||||||
| // for the root workspace, including allowed child types. | ||||||||||||||
| // +optional | ||||||||||||||
| Type *WorkspaceTypeReference `json:"type,omitempty"` | ||||||||||||||
| } | ||||||||||||||
|
|
||||||||||||||
| type WorkspaceRootStatus struct { | ||||||||||||||
| // Phase indicates the current state of the workspace root. | ||||||||||||||
| // +kubebuilder:default=Scheduling | ||||||||||||||
| Phase WorkspaceRootPhase `json:"phase,omitempty"` | ||||||||||||||
|
|
||||||||||||||
| // RootPath is the path identifier for this workspace tree root. | ||||||||||||||
| // This is the opaque identifier users will use to access this tree. | ||||||||||||||
| // Example: "a1b2c3d4" rather than "root:orgs:company-a" | ||||||||||||||
| RootPath string `json:"rootPath,omitempty"` | ||||||||||||||
|
|
||||||||||||||
| // URL is the base URL for accessing workspaces in this tree. | ||||||||||||||
| // +optional | ||||||||||||||
| URL string `json:"url,omitempty"` | ||||||||||||||
|
mjudeikis marked this conversation as resolved.
Outdated
|
||||||||||||||
|
|
||||||||||||||
| // Conditions represent the current state of the workspace root. | ||||||||||||||
| // +optional | ||||||||||||||
| Conditions conditionsv1alpha1.Conditions `json:"conditions,omitempty"` | ||||||||||||||
|
|
||||||||||||||
| // LogicalCluster is the name of the LogicalCluster backing this root. | ||||||||||||||
| // +optional | ||||||||||||||
| LogicalCluster string `json:"logicalCluster,omitempty"` | ||||||||||||||
| } | ||||||||||||||
|
|
||||||||||||||
| type WorkspaceRootPhase string | ||||||||||||||
|
|
||||||||||||||
| const ( | ||||||||||||||
| WorkspaceRootPhaseScheduling WorkspaceRootPhase = "Scheduling" | ||||||||||||||
| WorkspaceRootPhaseInitializing WorkspaceRootPhase = "Initializing" | ||||||||||||||
| WorkspaceRootPhaseReady WorkspaceRootPhase = "Ready" | ||||||||||||||
| WorkspaceRootPhaseDeleting WorkspaceRootPhase = "Deleting" | ||||||||||||||
| ) | ||||||||||||||
| ``` | ||||||||||||||
|
|
||||||||||||||
| ### Architecture | ||||||||||||||
|
|
||||||||||||||
| ``` | ||||||||||||||
| ┌─────────────────────────────────────────────────────────────────┐ | ||||||||||||||
| │ kcp cluster │ | ||||||||||||||
| ├─────────────────────────────────────────────────────────────────┤ | ||||||||||||||
| │ root (system workspace) │ | ||||||||||||||
| │ ├── orgs:workspaceroots (stores WorkspaceRoot objects) │ | ||||||||||||||
| │ │ ├── WorkspaceRoot/company-a → provisions "a1b2c3d4" │ | ||||||||||||||
| │ │ ├── WorkspaceRoot/company-b → provisions "x9y8z7w6" │ | ||||||||||||||
| │ │ └── WorkspaceRoot/company-c → provisions "m5n6o7p8" │ | ||||||||||||||
| │ └── ... │ | ||||||||||||||
| ├─────────────────────────────────────────────────────────────────┤ | ||||||||||||||
| │ a1b2c3d4 (company-a's root) ← independent tree │ | ||||||||||||||
| │ ├── team1 │ | ||||||||||||||
| │ │ ├── project1 │ | ||||||||||||||
| │ │ └── project2 │ | ||||||||||||||
| │ └── team2 │ | ||||||||||||||
| ├─────────────────────────────────────────────────────────────────┤ | ||||||||||||||
| │ x9y8z7w6 (company-b's root) ← independent tree │ | ||||||||||||||
| │ ├── engineering │ | ||||||||||||||
| │ └── sales │ | ||||||||||||||
| ├─────────────────────────────────────────────────────────────────┤ | ||||||||||||||
| │ m5n6o7p8 (company-c's root) ← independent tree │ | ||||||||||||||
| │ └── ... │ | ||||||||||||||
| └─────────────────────────────────────────────────────────────────┘ | ||||||||||||||
| ``` | ||||||||||||||
|
|
||||||||||||||
| ### User Experience | ||||||||||||||
|
|
||||||||||||||
| Users interact with their workspace tree using the opaque root identifier: | ||||||||||||||
|
|
||||||||||||||
| ```bash | ||||||||||||||
| # User from company-a sees: | ||||||||||||||
| $ kubectl ws . | ||||||||||||||
| Current workspace is 'a1b2c3d4:team1:project1'. | ||||||||||||||
|
|
||||||||||||||
| $ kubectl ws tree | ||||||||||||||
| a1b2c3d4 | ||||||||||||||
| ├── team1 | ||||||||||||||
| │ ├── project1 (current) | ||||||||||||||
| │ └── project2 | ||||||||||||||
| └── team2 | ||||||||||||||
|
|
||||||||||||||
| # Navigation works within the tree | ||||||||||||||
| $ kubectl ws use :a1b2c3d4:team2 | ||||||||||||||
| Current workspace is 'a1b2c3d4:team2'. | ||||||||||||||
|
|
||||||||||||||
| # Cannot navigate to sibling trees (no visibility, must have access to know the root path) | ||||||||||||||
| $ kubectl ws use :x9y8z7w6 | ||||||||||||||
| Error: access to workspace "x9y8z7w6" denied | ||||||||||||||
|
|
||||||||||||||
| # Kubeconfig references use opaque paths | ||||||||||||||
| $ kubectl config view | ||||||||||||||
| clusters: | ||||||||||||||
| - cluster: | ||||||||||||||
| server: https://kcp.example.com/clusters/a1b2c3d4:team1:project1 | ||||||||||||||
| name: workspace | ||||||||||||||
| ``` | ||||||||||||||
|
|
||||||||||||||
| ### Controller Behavior | ||||||||||||||
|
|
||||||||||||||
| The WorkspaceRoot controller is responsible for: | ||||||||||||||
|
|
||||||||||||||
| 1. **Scheduling**: Selecting a shard for the new root LogicalCluster. | ||||||||||||||
| 2. **Provisioning**: Creating the LogicalCluster with a unique base36 identifier. | ||||||||||||||
| 3. **Initialization**: Applying WorkspaceType configuration if specified. | ||||||||||||||
| 4. **Lifecycle Management**: Handling deletion with proper cleanup. | ||||||||||||||
|
|
||||||||||||||
| ``` | ||||||||||||||
|
ntnn marked this conversation as resolved.
Outdated
|
||||||||||||||
| WorkspaceRoot Created | ||||||||||||||
| │ | ||||||||||||||
| ▼ | ||||||||||||||
| ┌───────────────┐ | ||||||||||||||
| │ Scheduling │ ─── Select target shard | ||||||||||||||
| └───────┬───────┘ | ||||||||||||||
| │ | ||||||||||||||
| ▼ | ||||||||||||||
| ┌───────────────┐ | ||||||||||||||
| │ Initializing │ ─── Create LogicalCluster, apply type config | ||||||||||||||
| └───────┬───────┘ | ||||||||||||||
| │ | ||||||||||||||
| ▼ | ||||||||||||||
| ┌───────────────┐ | ||||||||||||||
| │ Ready │ ─── Tree is accessible | ||||||||||||||
| └───────────────┘ | ||||||||||||||
| ``` | ||||||||||||||
|
|
||||||||||||||
| ### Deletion Behavior | ||||||||||||||
|
|
||||||||||||||
| When a WorkspaceRoot is deleted: | ||||||||||||||
|
|
||||||||||||||
| 1. A finalizer prevents immediate deletion. | ||||||||||||||
| 2. All child workspaces within the tree are deleted recursively. | ||||||||||||||
| 3. The root LogicalCluster is deleted. | ||||||||||||||
| 4. The finalizer is removed and the WorkspaceRoot object is deleted. | ||||||||||||||
|
|
||||||||||||||
| This cascading deletion is guarded by finalizers on child workspaces to ensure | ||||||||||||||
| proper cleanup order. | ||||||||||||||
|
|
||||||||||||||
| ### API Location and RBAC | ||||||||||||||
|
|
||||||||||||||
| The WorkspaceRoot API is made available through APIExport/APIBinding: | ||||||||||||||
|
|
||||||||||||||
| 1. The `tenancy` APIExport in root workspace includes WorkspaceRoot. | ||||||||||||||
| 2. Administrators bind the tenancy API to workspaces where they want to allow | ||||||||||||||
| tree creation. | ||||||||||||||
| 3. No binding = no permission to create WorkspaceRoots. | ||||||||||||||
|
|
||||||||||||||
| This follows the existing pattern for Workspace API availability and provides | ||||||||||||||
| natural RBAC scoping. | ||||||||||||||
|
|
||||||||||||||
| ### Naming and Identifiers | ||||||||||||||
|
|
||||||||||||||
| - **User-provided names**: If `spec.name` is provided, it's used as metadata but | ||||||||||||||
| the root path is still a generated opaque identifier. | ||||||||||||||
| - **Generated identifiers**: Base36 encoded, providing sufficient entropy to | ||||||||||||||
| prevent guessing (e.g., `a1b2c3d4`, `x9y8z7w6`). | ||||||||||||||
|
mjudeikis marked this conversation as resolved.
Outdated
|
||||||||||||||
| - **Root workspace**: The system `root` workspace continues to exist for system | ||||||||||||||
| resources but organizations get their own trees. | ||||||||||||||
|
|
||||||||||||||
|
Comment on lines
+270
to
+276
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
The first point is already discussed above, the second point is irrelevant imho |
||||||||||||||
| ## Implementation Phases | ||||||||||||||
|
|
||||||||||||||
| ### Phase 1: Core API and Controller | ||||||||||||||
|
|
||||||||||||||
| 1. Define WorkspaceRoot CRD in `tenancy.kcp.io/v1alpha1`. | ||||||||||||||
| 2. Implement basic controller for lifecycle management. | ||||||||||||||
| 3. Basic scheduling (single shard initially). | ||||||||||||||
| 4. Integration with existing LogicalCluster provisioning. | ||||||||||||||
|
|
||||||||||||||
| ### Phase 2: WorkspaceType Integration | ||||||||||||||
|
|
||||||||||||||
| 1. Support WorkspaceType reference in WorkspaceRoot spec. | ||||||||||||||
| 2. Apply type configuration to root workspace. | ||||||||||||||
| 3. Inherit WorkspaceAuthenticationConfiguration. | ||||||||||||||
|
|
||||||||||||||
| ### Phase 3: Multi-Shard Support | ||||||||||||||
|
|
||||||||||||||
| 1. Shard selection for WorkspaceRoot placement. | ||||||||||||||
| 2. Cross-shard tree management. | ||||||||||||||
|
|
||||||||||||||
| ## Alternatives Considered | ||||||||||||||
|
|
||||||||||||||
| ### Alternative 1: Modify Workspace API with `--root` flag | ||||||||||||||
|
|
||||||||||||||
| ```bash | ||||||||||||||
| kubectl ws create bar --enter --root | ||||||||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||||||||||
| # Creates :bar as new root instead of :root:current:bar | ||||||||||||||
| ``` | ||||||||||||||
|
|
||||||||||||||
| **Rejected because:** | ||||||||||||||
| - Changes workspace behavior based on flags, making the API inconsistent. | ||||||||||||||
| - Breaks hierarchical semantics (`ws use ..` becomes ambiguous). | ||||||||||||||
| - Cannot scope permissions - anyone with Workspace create can create roots. | ||||||||||||||
| - Creates "soft-link" like behavior that breaks tree navigation. | ||||||||||||||
|
|
||||||||||||||
| ### Alternative 2: Remove `root` prefix entirely | ||||||||||||||
|
|
||||||||||||||
| Make the top-level workspace have no prefix, so workspaces appear as: | ||||||||||||||
| ``` | ||||||||||||||
| my-org:team:project # instead of root:my-org:team:project | ||||||||||||||
| ``` | ||||||||||||||
|
|
||||||||||||||
| **Rejected because:** | ||||||||||||||
| - Breaking change affecting all existing deployments. | ||||||||||||||
| - `root` is referenced in many places as `CoreRootCluster`. | ||||||||||||||
| - Doesn't solve the sibling visibility problem. | ||||||||||||||
|
|
||||||||||||||
| ### Alternative 3: Path translation/facade | ||||||||||||||
|
|
||||||||||||||
| Translate paths at the API layer so users see `a1b2c3:...` but internally | ||||||||||||||
| it's still `root:orgs:company-a:...`. | ||||||||||||||
|
|
||||||||||||||
| **Rejected because:** | ||||||||||||||
| - Security through obscurity - structure still exists. | ||||||||||||||
| - Complexity in maintaining translation layer. | ||||||||||||||
| - Doesn't provide true isolation. | ||||||||||||||
|
|
||||||||||||||
| ## References | ||||||||||||||
|
|
||||||||||||||
| - GitHub Issue: https://github.com/kcp-dev/kcp/issues/3716 | ||||||||||||||
| - Related KEP: [Decouple Logical Clusters from Hierarchy](decouple-logical-clusters-from-hierarchy.md) | ||||||||||||||
| - Related KEP: [Workspace Mounts](0002-proxy-workspace.md) | ||||||||||||||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.