Add KEP: ephemeral resources

[nfarhadian]

Proposes non-persisted resources served through kcp: an APIResourceSchema may declare spec.ephemeral, and the exposing APIExport nominates a webhook that answers create requests synchronously. Nothing reaches etcd.

Covers the webhook contract (EphemeralReview), the cross-shard constraint that rules out provider-supplied client certificates, why get cannot be served, and permission-claim handling.

[kcp-ci-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign xrstf for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kcp-ci-bot]

Hi @nfarhadian. Thanks for your PR.

I'm waiting for a kcp-dev member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[mjudeikis]

Few comments. I think API shape could be slightly different and it would simplify this a lot

[mjudeikis]

I read this and thinking outloud:

// ResourceSchemaStorage defines how the resource is stored.
//
// +kubebuilder:validation:XValidation:rule="has(self.crd) != has(self.virtual)",message="Exactly one of crd or virtual must be set"
type ResourceSchemaStorage struct {
	// CRD storage defines that this APIResourceSchema is exposed as
	// CustomResourceDefinitions inside the workspaces that bind to the APIExport.
	// Like in vanilla Kubernetes, users can then create, update and delete
	// custom resources.
	//
	// +optional
	CRD *ResourceSchemaStorageCRD `json:"crd,omitempty"`

	// Virtual storage defines that this APIResourceSchema is exposed as
	// a projection of the referenced resource inside the workspaces that
	// bind to the APIExport.
	//
	// +optional
	Virtual *ResourceSchemaStorageVirtual `json:"virtual,omitempty"`
}

we have this on the ResourceSchema storage. I think if we add one more Storage type "Ephemeral" into this with config to webhook, this plays very neat. Schema dont need even to be special (no Ephemeral{} hook/marker).

If APIExport find this storage type - it does what needs to be done?

[nfarhadian]

That's such a good idea, I will change it.

[nfarhadian]

I changed it

[mjudeikis]

This akk would go into ResourceSchemaStorage

[mjudeikis]

This feels bit all around the place. Can you simlify, what this section trying to explain?

[nfarhadian]

It is about why provider cannot simply give it's own client certificate to KCP using secret. I will change it.

[mjudeikis]

I think we need dryRun too to give signal for side effects...

[nfarhadian]

It's a good idea, but if it's "Ephemeral" nothing risky should be done with this...

My use cases for this type of resource is getting information, not doing something with it
For example:

• Bucket Info
• Get usage with custom time range
• Get list of accessible resources

[mjudeikis]

I think this needs more thinking. like FailurePolicy: ignore - what does it give?

[nfarhadian]

I was thinking returning 503 is ambiguous sometimes, because kcp is working but the webhook is not working. We have similar dynamic in kubernetes when operator is not working, no status means there isn't any response from operator.
But you are right, probably no one uses this, it adds complexity

[xrstf]

Tbh I don't like this approach at all. It strongly feels like re-inventing what VWs are already there for.

a client that bound the APIExport expects the resource in its own workspace, not at a separate VW URL. Since the same REST storage serves both, restricting to VWs buys nothing and costs the primary use case.

Reading URLs and connecting to other endpoints is a common thing in kcp, like in APIExportEndpointSlices.

The comparison with Kubernetes' built-in ephemeral resources is lacking, since those resources are served by Kubernetes internally. It doesn't have to call out to the wider internet to answer these. This feature here however would by design turn the kcp server into a synchronous proxy to random, potentially slow and failing backends, purely for a bit of perceived convenience, since every request to an ephemeral thing has to be forwarded.

[mjudeikis]

But you have to admit, this is useful feature. If somebody abuses it, it's their fault. Same as ValidationAdmissionWebhook is useful in kube but if abused its painful.

[gman0]

We have virtual resources (storage: virtual), where you can project ANY virtual workspace resources into ws?

[nfarhadian]

@gman0 You mean I create a workspace mount and set URL to my webhook and my webhook answers like kubernetes API. then I create an APIExport with storage: virtual
Am I getting your point right?

[gman0]

We could have a webhook virtual workspace & an endpoint slice for it, and that's it basically. You could check out the Replication VW as an example.

[ntnn]

We could have a webhook virtual workspace & an endpoint slice for it, and that's it basically. You could check out the Replication VW as an example.

I like the idea of APIWebhoookExport or similar that results in an APIExportES and then reference that with type:virtual.

[xrstf]

But you have to admit, this is useful feature.

I'm not denying that it sounds useful. But if that was the only metric we go by, we'd had cluster inheritance in kcp ;-)

What irks me is that I have bad feelings about turning kcp into an open HTTPS relay. How would we make sure this is not abused to DoS a shard by causing excessive, slow, large requests? Did we remember to forbid local/cluster-internal access to exfiltrate data from random pods?

If we want to configure an external backend, then that external backend is in fact already a virtual workspace, even if it's called a "webhook". And for proxying to virtual workspaces we generally use the front-proxy, not the shards directly. Ideally this feature makes the mapping known to the front-proxy, which can then react and proxy accordingly, instead of having the shards do it.

We wanted to move away from webhooks if possible (in general in the Kube world, see CEL), but this is adding more webhooks for the primary reason of "user convenience".

[ntnn]

What irks me is that I have bad feelings about turning kcp into an open HTTPS relay. How would we make sure this is not abused to DoS a shard by causing excessive, slow, large requests? Did we remember to forbid local/cluster-internal access to exfiltrate data from random pods?

Open implies that it goes to any - here the API owner has to specify the address and it can only go to said address.

And for proxying to virtual workspaces we generally use the front-proxy, not the shards directly.

Well, the shards have their own VW and the front-proxy mounts singleton/global virtual workspaces (or any http tbh).

Ideally this feature makes the mapping known to the front-proxy, which can then react and proxy accordingly, instead of having the shards do it.

This feature must exist in the virtual-workspaces because it's per APIExport; the front-proxy cannot route vw requests correctly.

We wanted to move away from webhooks if possible (in general in the Kube world, see CEL), but this is adding more webhooks for the primary reason of "user convenience".

I agree; I'd also prefer a simpler solution. As mentioned in the CC I'd prefer something that resembles KRO, but I think that will be hard to implement right, harder to use and make some use cases not possible.

Playing devils advocate - I wonder how much of ephemeral resources could already be implemented using mutating webhooks; it'd basically be the same just that the resource is persisted (and has the be created, then read and then deleted).

[mjudeikis]

I like the idea of APIWebhoookExport or similar that results in an APIExportES and then reference that with type:virtual.

This could work. We just need to hangle it right (allowed methods ,rbac, etc)

[xrstf]

Open implies that it goes to any - here the API owner has to specify the address and it can only go to said address.

Yes, but in an open platform, "API owner" is any random service provider. ;) Can this be restricted with RBAC, that someone can create an APIExport but not use webhook-based resources?

[ntnn]

Yes, but in an open platform, "API owner" is any random service provider. ;) Can this be restricted with RBAC, that someone can create an APIExport but not use webhook-based resources?

Ah :D Yes, it must be gated behind a feature flag and additional authz so not just anyone can create APIExportWebhooks.
We could do it similar to logical cluster migrations and have feature-gate + separate APIExport must be bound.

[mjudeikis]

We could have a webhook virtual workspace & an endpoint slice for it, and that's it basically. You could check out the Replication VW as an example.

so current limitation is that we dont replicate arbitrary APIExportEndpoint slices to cache servers. So resolution does not quite work. And if we try to piggyback on an existing object, it gets reconciled by default controllers.

Im checking code what needs to change, but building on VirtualResources flow is complicated/does-not-yet-works if you want BYO xEndpointSlice