Skip to content

feat: Add multi-part upload for large objects - #16374

Open
emadolsky wants to merge 1 commit into
grafana:mainfrom
emadolsky:emadolsky/implement-multi-part-upload-for-large-objects
Open

emadolsky wants to merge 1 commit into
grafana:mainfrom
emadolsky:emadolsky/implement-multi-part-upload-for-large-objects

Conversation

@emadolsky

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:
We have observed that bloom blocks are sometimes larger than the S3 single part size limit (5GB) which causes failure in putting the blocks in S3. This commit adds the functionality to use multi-part upload only if the size is larger than the limit.

Which issue(s) this PR fixes:
Fixes #13444

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@emadolsky
emadolsky requested a review from a team as a code owner February 19, 2025 13:56
@emadolsky
emadolsky force-pushed the emadolsky/implement-multi-part-upload-for-large-objects branch from 81a5c57 to d2803f8 Compare February 19, 2025 13:57
@emadolsky emadolsky changed the title Add multi-part upload for large objects feat: Add multi-part upload for large objects Feb 19, 2025
We have observed that bloom blocks are sometimes larger than the
S3 single part size limit (5GB) which causes failure in putting the
blocks in S3. This commit adds the functionality to use multi-part
upload only if the size is larger than the limit.
@emadolsky
emadolsky force-pushed the emadolsky/implement-multi-part-upload-for-large-objects branch from d2803f8 to e6a3c59 Compare February 19, 2025 13:59
@emadolsky

Copy link
Copy Markdown
Contributor Author

Hey @chaudum! Could you please have a look at this?

@chaudum

chaudum commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Since bloom filters are experimental and will be removed with the next major Loki release, I consider this feature as not relevant any more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Loki 3 Bloom compactor failed due to EntityTooLarge

2 participants