-
Notifications
You must be signed in to change notification settings - Fork 657
Add package staging schema and status support #10918
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: feature/staging
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,107 @@ | ||
| // Copyright (c) .NET Foundation. All rights reserved. | ||
| // Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information. | ||
|
|
||
| using System; | ||
| using System.ComponentModel.DataAnnotations; | ||
|
|
||
| namespace NuGet.Services.Entities | ||
| { | ||
| /// <summary> | ||
| /// Ephemeral staging metadata for a package in <see cref="PackageStatus.Staged"/>. The row is created when the | ||
| /// package is pushed to staging and hard-deleted when the package is promoted, deleted, or expires. | ||
| /// </summary> | ||
| /// <remarks> | ||
| /// <para> | ||
| /// Every staged package has a required <see cref="Owner"/>: ownership is stored here rather than derived from the | ||
| /// package registration's owners, so that a package with no <see cref="StagingGroup"/> is still attributable to an | ||
| /// account. This is an accounting concept, not an authorization one — the right to promote is still evaluated | ||
| /// against the package registration's owners. | ||
| /// </para> | ||
| /// <para> | ||
| /// This data is deliberately kept off the permanent <see cref="Package"/> row because it is meaningless once the | ||
| /// package leaves staging. The durable record of a promotion is <see cref="Package.ApproverUserKey"/>. | ||
| /// </para> | ||
| /// </remarks> | ||
| public class StagedPackage : IEntity | ||
| { | ||
| public const int MaxBlobPathLength = 512; | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the primary key for the entity. | ||
| /// </summary> | ||
| public int Key { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the key of the staged package. Unique: a package has at most one staging row. | ||
| /// </summary> | ||
| public int PackageKey { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the staged package. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Gets or sets? It should be a navigation property value of which defined by |
||
| /// </summary> | ||
| public virtual Package Package { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the key of the user or organization that owns this staged package. This is the account the | ||
| /// package is charged against for <see cref="User.StagingPackageLimit"/>, and it is not the same question as | ||
| /// who owns the package registration: promotion is still authorized against the registration's owners. | ||
| /// </summary> | ||
| public int OwnerKey { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the user or organization that owns this staged package. | ||
| /// </summary> | ||
| public virtual User Owner { get; set; } | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Package ownership right now happens on package registration level and there might be multiple owners. How that property fits into existing ownership setup? |
||
|
|
||
| /// <summary> | ||
| /// Gets or sets the key of the group this package belongs to, or <c>null</c> if the package is ungrouped. | ||
| /// When set, the group's owner must match <see cref="OwnerKey"/>. | ||
| /// </summary> | ||
| public int? StagingGroupKey { get; set; } | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can a package be staged to multiple groups at the same time? |
||
|
|
||
| /// <summary> | ||
| /// Gets or sets the group this package belongs to, or <c>null</c> if the package is ungrouped. | ||
| /// </summary> | ||
| public virtual StagingGroup StagingGroup { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets when the package entered staging. The timestamp is in UTC. | ||
| /// </summary> | ||
| public DateTime CreatedDate { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets how far validation has got for this package. A staged package stays in | ||
| /// <see cref="PackageStatus.Staged"/> throughout, so this is the only signal that it is promotable. | ||
| /// </summary> | ||
| public StagedValidationStatus ValidationStatus { get; set; } | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Don't we need 2 statuses here: one for regular package, another for symbols? |
||
|
|
||
| /// <summary> | ||
| /// Gets or sets the tracking ID of the validation set for this row's current content, assigned at push time | ||
| /// and rewritten whenever the content is replaced. Validation outcomes from any other set describe content | ||
| /// that has since been superseded and must be ignored, since a replace reuses the same package key. | ||
| /// </summary> | ||
| public Guid ValidationTrackingId { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the absolute deadline after which this package is eligible for cleanup. For grouped packages | ||
| /// this mirrors the group's expiration; for ungrouped packages it is set at push time and reset on replace. | ||
| /// The timestamp is in UTC. | ||
| /// </summary> | ||
| public DateTime ExpirationDate { get; set; } | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What happens on expiration? |
||
|
|
||
| /// <summary> | ||
| /// Gets or sets the path to the nupkg in the private staging container. The path embeds a GUID so that | ||
| /// deleting and re-uploading the same id and version cannot collide with the outgoing blob. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If deleting leaves a chance for the blob to stay, what's the plan to clean up the storage from stale entries? |
||
| /// </summary> | ||
| [Required] | ||
| [StringLength(MaxBlobPathLength)] | ||
| public string BlobPath { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the path to the companion snupkg in the private staging container, or <c>null</c> if no | ||
| /// symbol package has been staged. Set once symbol validation succeeds. | ||
| /// </summary> | ||
| [StringLength(MaxBlobPathLength)] | ||
| public string SnupkgBlobPath { get; set; } | ||
| } | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,28 @@ | ||
| // Copyright (c) .NET Foundation. All rights reserved. | ||
| // Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information. | ||
|
|
||
| namespace NuGet.Services.Entities | ||
| { | ||
| /// <summary> | ||
| /// The validation state of a package in <see cref="PackageStatus.Staged"/>. A staged package stays in | ||
| /// <see cref="PackageStatus.Staged"/> for its whole time in staging; this tracks how far validation got. | ||
| /// </summary> | ||
| public enum StagedValidationStatus | ||
| { | ||
| /// <summary> | ||
| /// Validation is in progress. This is the state a package is created in at push time. | ||
| /// </summary> | ||
| Validating = 0, | ||
|
|
||
| /// <summary> | ||
| /// Validation completed successfully. Only packages in this state may be promoted. | ||
| /// </summary> | ||
| Succeeded = 1, | ||
|
|
||
| /// <summary> | ||
| /// Validation failed. The package remains staged so the owner can inspect the failure, replace it, or | ||
| /// delete it, but it can never be promoted. | ||
| /// </summary> | ||
| Failed = 2, | ||
| } | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| // Copyright (c) .NET Foundation. All rights reserved. | ||
| // Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information. | ||
|
|
||
| using System; | ||
| using System.ComponentModel.DataAnnotations; | ||
|
|
||
| namespace NuGet.Services.Entities | ||
| { | ||
| /// <summary> | ||
| /// A table-as-queue of staging blobs awaiting deletion. Entries are enqueued in the same transaction that removes | ||
| /// a <see cref="StagedPackage"/> row, and drained later by the Gallery.Maintenance cleanup task. | ||
| /// </summary> | ||
| /// <remarks> | ||
| /// Blob deletion is deferred rather than performed inline so that promotion and delete requests do not block on | ||
| /// storage calls. This follows the existing PackageRevalidations table-as-queue pattern. | ||
| /// <para> | ||
| /// The queue is drained in primary key order, which the clustered key already provides, so | ||
| /// <see cref="CreatedDate"/> is deliberately not indexed. It exists for diagnosing a backed-up queue, not for | ||
| /// ordering it. | ||
| /// </para> | ||
| /// </remarks> | ||
| public class StagingBlobCleanup : IEntity | ||
| { | ||
| /// <summary> | ||
| /// Matches <see cref="StagedPackage.MaxBlobPathLength"/>: every path enqueued here comes from a | ||
| /// <see cref="StagedPackage"/>, so the two must not drift. | ||
| /// </summary> | ||
| public const int MaxBlobPathLength = StagedPackage.MaxBlobPathLength; | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the primary key for the entity. | ||
| /// </summary> | ||
| public int Key { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the path of the blob in the staging container that needs to be deleted. | ||
| /// </summary> | ||
| [Required] | ||
| [StringLength(MaxBlobPathLength)] | ||
| public string BlobPath { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets when this entry was enqueued. The timestamp is in UTC. | ||
| /// </summary> | ||
| public DateTime CreatedDate { get; set; } | ||
| } | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| // Copyright (c) .NET Foundation. All rights reserved. | ||
| // Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information. | ||
|
|
||
| using System; | ||
| using System.Collections.Generic; | ||
| using System.ComponentModel.DataAnnotations; | ||
|
|
||
| namespace NuGet.Services.Entities | ||
| { | ||
| /// <summary> | ||
| /// A named collection of staged packages belonging to a single owner. Groups let an author stage a set of | ||
| /// interdependent packages and promote them together, so consumers never observe a partially published set. | ||
| /// </summary> | ||
| /// <remarks> | ||
| /// <para> | ||
| /// Groups are optional. A staged package with no group has a <c>null</c> <see cref="StagedPackage.StagingGroupKey"/>. | ||
| /// A staged package belongs to at most one group, enforced by the unique index on | ||
| /// <see cref="StagedPackage.PackageKey"/>. | ||
| /// </para> | ||
| /// <para> | ||
| /// Groups are hard-deleted. Deleting a group removes its <see cref="StagedPackage"/> rows and enqueues their | ||
| /// blobs to <see cref="StagingBlobCleanup"/> in the same transaction, so the group row has no remaining | ||
| /// dependents and there is nothing left for the cleanup job to defer. | ||
| /// </para> | ||
| /// </remarks> | ||
| public class StagingGroup : IEntity | ||
| { | ||
| public const int MaxIdLength = 64; | ||
| public const int MaxNameLength = 256; | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the primary key for the entity. | ||
| /// </summary> | ||
| public int Key { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the user-specified slug for this group: lowercase alphanumeric characters and dashes. | ||
| /// Unique per owner, not globally. | ||
| /// </summary> | ||
| [Required] | ||
| [StringLength(MaxIdLength)] | ||
| public string Id { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the key of the user or organization that owns this group. | ||
| /// </summary> | ||
| public int OwnerKey { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the user or organization that owns this group. Any user with push permissions under this | ||
| /// owner may add packages to the group, promote it, or delete it. | ||
| /// </summary> | ||
| public virtual User Owner { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the display name for this group. Set to <see cref="Id"/> when the author does not supply one. | ||
| /// </summary> | ||
| [Required] | ||
| [StringLength(MaxNameLength)] | ||
| public string Name { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets when the group was created. The timestamp is in UTC. | ||
| /// </summary> | ||
| public DateTime CreatedDate { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the absolute deadline after which this group is eligible for cleanup. Set on creation and | ||
| /// reset on every push to the group. The timestamp is in UTC. | ||
| /// </summary> | ||
| public DateTime ExpirationDate { get; set; } | ||
|
|
||
| /// <summary> | ||
| /// Gets or sets the packages staged in this group. | ||
| /// </summary> | ||
| public virtual ICollection<StagedPackage> StagedPackages { get; set; } | ||
| } | ||
| } |
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we are not going to have multiple entries in this table for an entry in Packages table, can we just use
PackageKeyhere as primary key and drop this column?Is there a scenario where look up in this table would be done NOT by package key?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm still refining the design so hold off on reviewing until I republish the PR.