Skip to content

feat: add bulk import DTO request/response and Status-returning BulkImport overloads - #613

Open
yhmo wants to merge 1 commit into
milvus-io:masterfrom
yhmo:ma
Open

yhmo wants to merge 1 commit into
milvus-io:masterfrom
yhmo:ma

Conversation

@yhmo

@yhmo yhmo commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Mirror the Java SDK bulk-writer request hierarchy for the C++ milvus::BulkImport REST utility. Currently the C++ bulk-import methods take scattered scalar parameters; this change introduces DTO-style request/response types matching the SDK's V2 request/response convention.

What's new

  • src/include/milvus/request/import/ request DTOs (with impls under src/impl/request/import/):
    • BaseImportRequest (CRTP template: ApiKey, Options) → MilvusImportRequest, VolumeImportRequest, CloudImportRequest
    • BaseDescribeImportRequest → MilvusDescribeImportRequest, CloudDescribeImportRequest
    • BaseListImportJobsRequest → MilvusListImportJobsRequest, CloudListImportJobsRequest
  • src/include/milvus/response/import/BulkImportResponse wrapping the raw REST JSON envelope (RawJson, Code, Message, Data, JobId).
  • BulkImport::CreateImportJobs/ListImportJobs/GetImportJobProgress/CommitImport/AbortImport overloads accepting the request DTOs and returning Status with a BulkImportResponse out-parameter. The legacy nlohmann::json overloads remain unchanged.

Notes

  • Base request classes use the CRTP pattern (like DBRequestBase<T>) so fluent WithXxx setters chain across base/derived boundaries.
  • DTO describe/commit/abort payloads use the server jobId field (matching the v2 REST JobIDReq contract and the Java SDK); the legacy GetImportJobProgress keeps its original jobID payload.
  • The DTO API key is sent via the Authorization: Bearer header only, consistent with the existing implementation.

Validation

  • 938 unit tests + 338 mocked tests pass (includes 8 new BulkImportTest.Dto* cases).
  • cpplint passes; clang-format-14 passes on all touched files.

Copilot AI balanced review requested due to automatic review settings October 9, 2026 03:10
@sre-ci-robot

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: yhmo

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mergify

mergify Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

  • Queue this pull request

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Server failures can return success statuses, exceptions can escape, and the public CRTP and aggregate-header contracts need correction.

7 open findings
What changed in this PR

Adds DTO-based bulk-import REST APIs aligned with V2 SDK conventions.

Changes:

  • Adds Milvus, Volume, and Cloud request DTO hierarchies.
  • Adds a JSON-backed bulk-import response DTO and Status overloads.
  • Adds local HTTP-server unit coverage for request serialization and routing.
File Description
src/​include/​milvus/​BulkImport.h Declares DTO-based overloads.
src/​impl/​BulkImport.cpp Implements shared REST handling.
src/​include/​milvus/​request/​import/​BaseImportRequest.h Defines common import fields.
src/​include/​milvus/​request/​import/​BaseDescribeImportRequest.h Defines common job-operation fields.
src/​include/​milvus/​request/​import/​BaseListImportJobsRequest.h Defines common list fields.
src/​include/​milvus/​request/​import/​MilvusImportRequest.h Declares Milvus import DTO.
src/​impl/​request/​import/​MilvusImportRequest.cpp Serializes Milvus imports.
src/​include/​milvus/​request/​import/​VolumeImportRequest.h Declares volume import DTO.
src/​impl/​request/​import/​VolumeImportRequest.cpp Serializes volume imports.
src/​include/​milvus/​request/​import/​CloudImportRequest.h Declares cloud import DTO.
src/​impl/​request/​import/​CloudImportRequest.cpp Serializes cloud imports.
src/​include/​milvus/​request/​import/​MilvusDescribeImportRequest.h Declares Milvus job DTO.
src/​impl/​request/​import/​MilvusDescribeImportRequest.cpp Serializes Milvus job operations.
src/​include/​milvus/​request/​import/​CloudDescribeImportRequest.h Declares cloud job DTO.
src/​impl/​request/​import/​CloudDescribeImportRequest.cpp Serializes cloud job operations.
src/​include/​milvus/​request/​import/​MilvusListImportJobsRequest.h Declares Milvus list DTO.
src/​impl/​request/​import/​MilvusListImportJobsRequest.cpp Serializes Milvus list requests.
src/​include/​milvus/​request/​import/​CloudListImportJobsRequest.h Declares cloud list DTO.
src/​impl/​request/​import/​CloudListImportJobsRequest.cpp Serializes cloud list requests.
src/​include/​milvus/​response/​import/​BulkImportResponse.h Declares response wrapper.
src/​impl/​response/​import/​BulkImportResponse.cpp Implements response accessors.
test/​ut/​TestBulkImport.cpp Tests DTO routing and payloads.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/impl/BulkImport.cpp
Comment thread src/impl/BulkImport.cpp Outdated
Comment thread src/impl/BulkImport.cpp Outdated
Comment thread src/include/milvus/BulkImport.h
Comment thread src/include/milvus/request/import/BaseDescribeImportRequest.h Outdated
Comment thread src/include/milvus/request/import/BaseImportRequest.h Outdated
Comment thread src/include/milvus/request/import/BaseListImportJobsRequest.h Outdated
@mergify mergify Bot added the ci-passed label Oct 9, 2026
@codecov

codecov Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 77.70419% with 101 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.42%. Comparing base (a0592aa) to head (21d0bc3).
⚠️ Report is 167 commits behind head on master.

Files with missing lines Patch % Lines
src/impl/request/import/CloudImportRequest.cpp 60.00% 40 Missing ⚠️
src/impl/request/import/VolumeImportRequest.cpp 78.18% 12 Missing ⚠️
...impl/request/import/CloudListImportJobsRequest.cpp 79.16% 10 Missing ⚠️
src/impl/BulkImport.cpp 84.61% 8 Missing ⚠️
...impl/request/import/CloudDescribeImportRequest.cpp 78.94% 8 Missing ⚠️
src/impl/request/import/MilvusImportRequest.cpp 78.37% 8 Missing ⚠️
...mpl/request/import/MilvusListImportJobsRequest.cpp 80.95% 4 Missing ⚠️
src/impl/response/import/BulkImportResponse.cpp 84.00% 4 Missing ⚠️
...mpl/request/import/MilvusDescribeImportRequest.cpp 81.81% 2 Missing ⚠️
.../milvus/request/import/BaseDescribeImportRequest.h 88.88% 2 Missing ⚠️
... and 2 more
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           master     #613       +/-   ##
===========================================
+ Coverage   53.47%   89.42%   +35.95%     
===========================================
  Files          52      413      +361     
  Lines        4432    17280    +12848     
  Branches        0     1881     +1881     
===========================================
+ Hits         2370    15453    +13083     
+ Misses       2062     1826      -236     
- Partials        0        1        +1     
Files with missing lines Coverage Δ
src/include/milvus/BulkImport.h 100.00% <100.00%> (ø)
src/include/milvus/MilvusClientV2.h 33.33% <ø> (ø)
...milvus/request/import/CloudDescribeImportRequest.h 100.00% <100.00%> (ø)
...include/milvus/request/import/CloudImportRequest.h 100.00% <100.00%> (ø)
...milvus/request/import/CloudListImportJobsRequest.h 100.00% <100.00%> (ø)
...ilvus/request/import/MilvusDescribeImportRequest.h 100.00% <100.00%> (ø)
...nclude/milvus/request/import/MilvusImportRequest.h 100.00% <100.00%> (ø)
...ilvus/request/import/MilvusListImportJobsRequest.h 100.00% <100.00%> (ø)
...nclude/milvus/request/import/VolumeImportRequest.h 100.00% <100.00%> (ø)
...nclude/milvus/response/import/BulkImportResponse.h 100.00% <100.00%> (ø)
... and 12 more

... and 408 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread src/impl/BulkImport.cpp Outdated
Comment thread src/include/milvus/BulkImport.h Outdated
Comment thread src/include/milvus/BulkImport.h Outdated
…mport overloads

Mirror the Java SDK bulk-writer request hierarchy: BaseImportRequest
(MilvusImportRequest/VolumeImportRequest/CloudImportRequest),
BaseDescribeImportRequest (Milvus/Cloud), and BaseListImportJobsRequest
(Milvus/Cloud), plus a BulkImportResponse that wraps the raw REST JSON.

Add CreateImportJobs/ListImportJobs/GetImportJobProgress/CommitImport/
AbortImport overloads that accept the request DTOs and return Status with a
BulkImportResponse out-parameter; the legacy nlohmann::json overloads stay
unchanged. The DTO describe/commit/abort payloads use the server jobId field.

DTO overloads surface server rejections through a nonzero envelope code,
classify transport vs HTTP failures, never leak exceptions, and the new DTOs
are reachable from the MilvusClientV2 aggregate header. CRTP base request
constructors are protected so they cannot be instantiated directly.

Normalize the legacy GetImportJobProgress payload to the server jobId field,
route the DTO GetImportJobProgress through /describe, and let describe/commit/
abort target a non-default database via the DB-Name header (BaseDescribeImportRequest
gains a database name). The DTO template wrappers delegate to private
non-template implementations in BulkImport.cpp so the header carries no REST
endpoint strings; the HTTP helpers are file-local to the implementation.

Signed-off-by: yhmo <yihua.mo@zilliz.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants