Skip to content

[glance] Comprehensive v2 API rate limiting (fix action keys + watcher normalization) - #12540

Open
rajivmucheli wants to merge 2 commits into
masterfrom
feature/glance-ratelimit-comprehensive
Open

[glance] Comprehensive v2 API rate limiting (fix action keys + watcher normalization)#12540
rajivmucheli wants to merge 2 commits into
masterfrom
feature/glance-ratelimit-comprehensive

Conversation

@rajivmucheli

Copy link
Copy Markdown
Contributor

Summary

  • Add rate-limit-middleware to all 24 Glance v2 API endpoints with per-project and global limits
  • Fix watcher path_keywords: add namespaces so metadef namespace names are normalized to namespace in the CADF URI (without this, sub-resource rules never matched)
  • Fix CADF action mismatches in ratelimit config: collection GETs generate read/list, not read; corrected for info/stores, metadefs/namespaces, metadefs/*/tags, schemas, and resource_types endpoints

Validation

Tested on QA using monsoon3/cc-demo credentials with production limits (300-1000r/m). HTTP 429 (Too Many Requests) is returned by the middleware when a project exceeds its per-minute limit. Each test sent a parallel burst larger than the configured limit and counted the 429 responses. Format: <429s received>/<total sent> (limit).

Endpoint 429s/Total Limit
images_list 100/700 600r/m
images_get 100/700 600r/m
images_members_list 50/350 300r/m
images_locations 49/350 300r/m
images_file_head 398/700 300r/m
tasks_list 50/350 300r/m
schemas_image 98/1100 1000r/m
schemas_images 100/1100 1000r/m
schemas_member 100/1100 1000r/m
schemas_members 99/1100 1000r/m
schemas_task 100/1100 1000r/m
schemas_tasks 100/1100 1000r/m
schemas_metadef_ns 99/1100 1000r/m
info_stores 49/350 300r/m
info_import 50/350 300r/m
info_usage 50/350 300r/m
metadefs_namespaces 49/350 300r/m
metadefs_namespace 50/350 300r/m
metadefs_properties 50/350 300r/m
metadefs_objects 48/350 300r/m
metadefs_tags 50/350 300r/m
metadefs_resource_types 50/350 300r/m
metadefs_ns_resource_types 50/350 300r/m
cache_list 250/350 100r/m

24/24 PASS

This commit extends the Glance rate limit configuration to cover all
API v2 endpoints, providing better protection against API abuse.

Changes:
- Add global rate limits for tasks, schemas, and metadefs endpoints
- Add local rate limits for:
  - Image file upload/download operations
  - Image stage and import workflow
  - Image actions (deactivate/reactivate)
  - Image members (sharing)
  - Image tags and locations
  - Tasks API (list, read, create, delete)
  - Schemas API (all schema endpoints)
  - Discovery/Info APIs (import, stores, usage)
  - Metadef namespaces, properties, objects, tags
  - Metadef resource types
  - Cache management endpoints
  - Multi-store operations

Rate limit values:
- Read operations: 300-600r/m per project
- Write operations: 50-100r/m per project
- Schemas (cacheable): 1000r/m per project
- Global limits: 2000-5000r/m across all projects

This ensures all Glance API endpoints are protected from excessive
requests while maintaining reasonable limits for normal operations.
@rajivmucheli
rajivmucheli force-pushed the feature/glance-ratelimit-comprehensive branch from 07a5ebb to 8dd39c5 Compare August 11, 2026 05:12
1. _watcher.yaml.tpl: add 'namespaces' to path_keywords so metadef
   namespace names are normalized to 'namespace' in the CADF URI instead
   of being treated as literal path segments (which caused no rate limit
   rule to ever match for metadef sub-resources)

2. _ratelimit.yaml.tpl: fix CADF action mismatches discovered via watcher
   middleware introspection:
   - schemas/*: add read/list (plural schema paths generate read/list)
   - info/stores: read -> read/list
   - metadefs/namespaces: read+list -> read/list (consolidated)
   - metadefs/namespaces/namespace/properties: add read/list
   - metadefs/namespaces/namespace/objects: add read/list
   - metadefs/namespaces/namespace/tags: read+list -> read/list
   - metadefs/resource_types: read+list -> read + read/list
   - metadefs/namespaces/namespace/resource_types: add read/list

QA validation (monsoon3/cc-demo, production limits 300-1000r/m):
24/24 endpoints confirmed rate-limited. HTTP 429 (Too Many Requests) is
returned by the middleware when a project exceeds its per-minute limit.
Each test sent a parallel burst larger than the configured limit and
counted the 429 responses. Format: "<429s received>/<total sent> (limit)".

  images_list               100/700  429s  (600r/m)
  images_get                100/700  429s  (600r/m)
  images_members_list        50/350  429s  (300r/m)
  images_locations           49/350  429s  (300r/m)
  images_file_head          398/700  429s  (300r/m)
  tasks_list                 50/350  429s  (300r/m)
  schemas_image              98/1100 429s  (1000r/m)
  schemas_images            100/1100 429s  (1000r/m)
  schemas_member            100/1100 429s  (1000r/m)
  schemas_members            99/1100 429s  (1000r/m)
  schemas_task              100/1100 429s  (1000r/m)
  schemas_tasks             100/1100 429s  (1000r/m)
  schemas_metadef_ns         99/1100 429s  (1000r/m)
  info_stores                49/350  429s  (300r/m)
  info_import                50/350  429s  (300r/m)
  info_usage                 50/350  429s  (300r/m)
  metadefs_namespaces        49/350  429s  (300r/m)
  metadefs_namespace         50/350  429s  (300r/m)
  metadefs_properties        50/350  429s  (300r/m)
  metadefs_objects           48/350  429s  (300r/m)
  metadefs_tags              50/350  429s  (300r/m)
  metadefs_resource_types    50/350  429s  (300r/m)
  metadefs_ns_resource_types 50/350  429s  (300r/m)
  cache_list                250/350  429s  (100r/m)
@rajivmucheli
rajivmucheli force-pushed the feature/glance-ratelimit-comprehensive branch from 8dd39c5 to 9f70dea Compare August 11, 2026 05:14
@sapcc-bot

Copy link
Copy Markdown
Contributor

Failed to validate the helm chart. Details. Readme.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants